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FOREWORD 


The  Guidelines  for  Conducting  a  Training  Effectiveness  Evaluation  (TEE), 
Volumes  1-3  were  developed  in  response  to  Army  Training  Study  HRN  79-269, 
entitled  Methodology  for  Training  Effectiveness  Analysis  (TEA). 

The  three  volumes  of  the  Guidelines  explain  how  to  use  the  TEE  methodol¬ 
ogy  to  evaluate  a  course  of  instruction  and  to  formulate  revisions  for  cor¬ 
recting  training  deficiencies,  in  ongoing  courses  and  in  training  for 
operational  tests  (OT's)  of  new  or  improved  equipment.  Both  product  analyses 
of  training  materials  and  process  analyses  of  training  methods  are  covered  by 
the  Guidelines.  Volumes  I  and  III  are  directed  toward  the  TEE  analyst  and 
associate  analysts  and  explain  how  to  conduct  a  TEE  and  how  to  remedy  defi¬ 
ciencies,  respectively.  Volume  II  addresses  the  needs  of  the  TEE  data  col¬ 
lector. 
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INTRODUCTION 


l.l  Background 

The  Army  Research  Institute  Field  Unit,  Fort  Bliss,  contracted  Calspan 
Corporation's  Advanced  Technology  to  develop  a  set  of  guidelines  and  data 
collection  procedures  and  instruments  for  conducting  a  Training  Effectiveness 
Evaluation  (TEE)  and  guidelines  for  modifying  deficient  aspects  of  training 
which  have  been  identified  by  the  TEE. 

The  TEE  system  resulting  from  this  effort  is  intended  to  be  particularly 
applicable  to  Army  Air  Defense  training,  but  should  also  have  utility  in 
evaluating  and  revising  any  Army  training  course  or  program. 

This  volume  Guidelines  for  Conducting  a  Training  Effectiveness 
Evaluation,  Volume  I:  User's  Guide  for  Revising  Training  Program 
Deficiencies,  provides  a  set  of  job  aids  for  modifying  Army  training  programs 
which  have  been  found  to  be  deficient  as  a  result  of  a  TEE. 

Earlier  published  parts  of  the  TEE  Ssytem  include  the  following: 

•  Guidelines  For  Conducting  a  Training  Effectiveness  Evaluation  - 
Volume  I:  TEE  Evaluator's  Handbook 

•  Guidelines  For  Conducting  a  Training  Effectiveness  Evaluation  - 
Volume  II:  Data  Collector's  Manual 

The  scope  of  a  TEE  project  includes  collecting  data  on  a  training  event 
for  the  purpose  of  identifying  failures  in  task  performance  by  trainees  (e.g., 
performance  discrepancies)  or  in  the  instructional  system  itself  (training 
deficiencies).  The  TEE  methodology  provides  a  structural  procedure  for 
observing  training  and  for  assessing  training  packages  in  terms  of  their 
conformance  with  the  Instructional  Systems  Development  procedures.  Data  on 
both  the  training  process,  collected  through  actual  observations  of  training, 
and  on  training  products  or  materials,  are  included  in  the  TEE.  Greater 
detail  about  the  procedures  for  conducting  a  TEE  and  its  specific  components 
can  be  found  in  the  Guidelines  for  Conducting  a  Training  Effectiveness 
Evaluation  -  Volume  I;  TEE  Evaluator's  Handbook. 


1.2  Purpose  and  Scope 

The  Guidelines  for  Conducting  a  Training  Effectiveness  Evaluation, 
Volume  I:  User's  Guide  for  Revising  Training  Program  Deficiencies,  addresses 
the  correction  of  product  and  process  deficiencies  which  are  discovered  as  a 
result  of  a  TEE.  As  was  the  case  with  the  TEE  guidelines,  the  breadth  of 
the  topics  covered  and  the  more  specific  steps  recommended  for  training 
program  modifciation  are  firmly  based  in  the  evaluation  literature.  The  major 
sources  of  reference  included  Phase  D  of  the  Harless  Guidelines  (Kristiansen, 
1981),  the  Methodology  for  Correcting 


Deficiencies  in  Training  Programs  (Seville  Project  Report  80-08),  and  other 
relevant  published  literature. 

The  purpose  of  this  manual  is  to  provide  TEE  analysts  with  guidelines 
for  suggesting  detailed  solutions  for  modifying  training  programs  to  increase 
training  efficiency  and  effectiveness,  with  respect  to  both  trainee 
performances  and  the  training  system.  For  ease  of  use,  the  solutions  have 
been  categorized  into  four  global  functional  aspects  of  the  training  program. 
These  are: 

1.  Training  environment  -  includes  activities  such  as  training  plan 
implementation,  motivational  components,  and  macro  aspects  of 
training  like  training  duration,  adequacy  of  equipment,  etc. 

2.  Presentation  -  covers  topics  related  to  the  adequacy,  sequence 
and  comprehensibility  of  a  course,  including  the  use  of  media  and 
visuals. 

3.  Practice /Demonstration  -  includes  process  and  product 
discrepancies  related  to  the  learning  and  practice  of  trainees. 

4.  Testing  -  includes  apsects  of  the  testing  activities  such  as 
adequacy  and  realism. 

Within  each  of  these  four  training  components,  specific  subtopics  and  detailed 
problems  are  discussed. 

1.3  Use  of  the  Guidelines 

The  manual  has  been  developed  as  a  part  of  the  TEE  system  to  aid  the 
analyst  in  the  progression  from  training  program  evaluation  to  revisions  of 
training  products  and  processes  that  will  increase  training  effectiveness.  Each 
of  the  problems  addressed  has  been  cross-referenced  with  TEE  worksheets. 
These  worksheets  point  to  specific  discrepancies  or  deficiencies  and  thus  serve 
as  inputs  for  identifying  potential  solutions.  In  addition,  the  problems  have 
been  keyed  to  particular  TEE  questions.  (For  convenience  these  questions  are 
reproduced  in  Appendix  A).  Since  the  questions  are  the  meat  of  the  TEE 
system,  it  is  natural  that  the  fixes  recommended  in  these  Guidelines  should  be 
directly  related  to  them. 
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TRAINING  ENVIRONMENT 


2.1  TRAINING  ENVIRONMENT:  Training  Site 

THE  PROBLEM.  Environmental  factors  in  the  training  site  were  such  as  to 
inhibit  training  effectiveness.  That  is,  the  site  itself  does  not  provide  enough 
light,  or  is  unbearably  hot  or  cold.  At  its  very  worst,  the  training  site  should 
at  least  not  detract  from  soldier  learning  and  retention. 

EVIDENCE  for  a  deficient  training  site  will  come  from  the  following 
worksheets: 

Cl.l  Observation  of  Classroom  Instruction 
Cl.2  Observation  of  Demonstrations 
Cl. 3  Observation  of  Isolated  Practice 
C1.4  Observation  of  a  Performance  Test 
C1.5  Observation  of  a  Written  Test 
C1.6  Observation  of  an  Oral  Test 
C1.7  Observation  of  Integrated  Practice 
C1.8  Observation  of  Individual  Study 

Question  85.  Is  there  enough  space  for  all  of  the  trainees? 

% 

Question  86.  Is  the  instruction  free  of  distractions? 

Question  87.  Is  the  lighting  appropriate  for  the  training  situation? 
Question  88.  Is  the  temperature  appropriate  for  the  training  situaton? 
RECOMMENDATIONS  for  training  program  modification. 

2.1.1  Not  Enough  Space  for  all  of  the  Trainees  (Question  85). 

There  just  was  not  enough  space  to  accommodate  all  the  trainees.  The 
training  site  should  provide  enough  space  so  that  each  trainee  can  comfortably 
see  everything  that  goes  on.  No  trainee  should  be  screened  by  another 
trainee  or  should  have  to  strain  to  get  a  good  view.  If  the  training  site  is 
too  small,  a  new  site  should  be  obtained  for  future  training  that  eliminates 
the  space  problems  of  the  current  site.  If  a  new  site  cannot  be  obtained, 
fewer  trainees  per  instructor  will  "expand”  the  space  available  at  the  current 
site. 

2.1.2  Training  Site  Too  Noisy  (Question  86). 

Excessive  noise  was  noted  as  a  serious  distraction  at  the  training  site.  Motor 
pools  are  a  good  example  of  this.  The  argument  is  often  advanced  that, 
since  trainees  may  have  to  perform  on  the  job  in  a  noisy  environment,  they 
should  learn  in  a  noisy  environment.  This  is  not  true.  Trainees  should  learn 


to  perform  tasks  in  sites  that  are  as  free  of  distractions  as  possible.  After 
task  proficiency  is  gained,  they  can  practice  in  a  noisy  environment.  The 
training  site  should  be  selected  so  as  to  enhance  training.  Noisy  environments 
hinder  effective  training.  If  trainees  cannot  readily  hear  the  instruction, 
effectiveness  (in  terms  of  trainee  performance)  and  efficiency  (in  terms  of 
time  to  proficiency)  both  suffer.  If  the  training  site  is  just  too  noisy,  a  new 
site  should  be  obtained  for  future  training  that  eliminates  the  noise  problems 
of  the  current  site. 


2.1.3  Not  Enough  Light  at  the  Training  Site  (Question  87). 

The  training  site  was  so  dark  that  the  trainees  had  difficulty  seeing  what  was 
going  on.  Each  trainee  should  be  able  to  see  clearly  everything  that  goes  on 
in  training.  The  argument  is  often  advanced  that,  since  trainees  may  have  to 
perform  on  the  job  under  conditions  of  reduced  visibility,  they  should  learn 
under  conditions  of  reduced  visibility.  This  is  not  true.  Trainees  should  learn 
to  perform  tasks  in  sites  that  are  as  free  of  distractions  as  possible.  After 
task  proficiency  is  gained,  then  they  can  practice  unde  conditions  of  reduced 
visibility.  If  the  training  site  is  too  dark,  a  new  site  should  be  obtained  for 
future  training  that  eliminates  the  lighting  problems  of  the  current  site,  or,  if 
possible,  the  lighting  in  the  current  site  should  be  enhanced. 

2.1.4  Temperature  Inappropriate  for  the  Training  Situation  (Question  83). 

The  trainees  were  so  warm  that  they  had  trouble  concentrating  on  the 
instruction,  or  so  cold  that  they  were  more  interested  in  keeping  warm  than 
in  paying  attention  to  the  instructor.  The  training  environment  should  aid 
learning  and  retention.  The  training  manager,  should  take  whatever  steps  are 
necessary  to  secure  a  favorable  training  environment  where  temperature  is  not 
extreme.  When^an  unfavorable  training  environment  must  be  used,  anticipate 
learning  and  retention  problems  and  take  steps,  in  advance,  to  offset  the 
effects  of  the  environment.  Such  steps  may  include  shorter  training  sessions 
with  more  breaks  in  cases  of  excessive  temperature. 

Bad  weather  may  also  have  affected  the  conduct  of  training.  Training  must 
adjust  to  inclement  weather.  If  the  weather  is  bad  enough  to  affect  training, 
training  should  be  postponed  or  moved  to  a  sheltered  site.  Since  this  is 
seldom  feasible,  a  more  realistic  solution  is  to  modify  the  training  process  to 
offset,  as  much  as  possible,  the'effects  of  the  weather  on  training.  The 
problem,  of  course,  is  that  when  trainees  are  uncomfortable,  they  attend 
more  to  their  comfort  than  the  training.  Do  not  confuse  training  with  the 
practice  of  tasks  already  learned.  Once  tasks  are  learned,  they  can  be 
practiced  in  bad  weather.  Training  time  during  bad  weather  should  be 
lengthened,  not  shortened.  The  amount  of  time  any  trainee  is  exposed  to  the 
weather  should  be  shortened.  Sheltered  sites  should  be  set  up  for  concurrent 
training  and  should  be  manned  by  instructors.  With  such  an  arrangement, 
students  can  be  trained  during  several  short  sessions  in  the  weather  and  can 
practice  associated  tasks  in  the  sheltered  site.  Both  trainees  and  instructors 
can  be  shuttled  back  and  forth.  If  none  of  these  solutions  are  feasible, 
remedial  training  needs  to  be  scheduled. 


2.2  TRAINING  ENVIRONMENT:  Implementation  of  Training  Plan 

THE  PROBLEM.  The  training  that  was  actually  conducted  was  not  a  faithful 
implementation  of  the  training  plan  (lesson  plan  or  instructor  guide).  The 
lesson  plan  should  lay  out  the  training  activities  to  be  completed  in  enough 
detail  for  any  instructor-trained  subject  matter  expert  to  follow.  The 
instructor  should  follow  this  prescription  for  training  this  task. 

EVIDENCE  for  an  unsound  implementation  of  the  training  plan  will  come  from 
the  following  worksheets: 

Cl.l  Observation  of  Classroom  Instruction 

C1.2  Observation  of  Demonstrations 

C1.3  Observation  of  Isolated  Practice 

C1.7  Observation  of  Integrated  Practice 

Question  82.  Does  the  instructor  follow  the  methods  in  the  Instructor 
Guide? 

Question  83.  Does  the  instructor  teach  all  of  the  content  in  the 
LESSON  materials? 


Question  84.  Did  the  instructor  limit  his  teaching  to  the  content  in  the 
LESSON  materials? 

RECOMMENDATIONS  for  training  program  modification. 

* 

2.2.1  Instructor  Did  Not  Follow  the  Instructor  Guide  (Question  82). 

The  instructor  did  not  follow  the  training  activities  in  the  lesson  plan  or 
Instructor  Guide.  It  is  assumed  that  the  Instructor  Guide  provides  a  good 
training  presentation  as  defined  in  Section  3  of  this  User's  Guide. 

If  it  does  not,  then  the  Instructor  Guide  itself  needs  to  be  rewritten.  The 
instructor  should  follow  the  Instructor  Guide.  If  he  or  she  feels  that  the 
Instructor  Guide  is  inaccurate  or  does  not  provide  good  training,  then  he  or 
she  should  take  steps  to  have  it  modified.  The  solution  to  an  inaccurate 
Instructor  Guide  is  not  just  to  stop  following  it.  If  the  instructor  has 
difficulty  following  the  Instructor  Guide  because  he  or  she  has  not  been 
"trained  to  train,"  instructor  training  should  be  immediately  scheduled. 


If  the  audiovisual  materials,  charts,  mock-ups,  handouts,  manuals,  or  other 
training  aids  or  equipment  called  for  in  the  Instructor  Guide  were  not  used, 
find  out  if  they  were  available.  These  materials  should  be  collected  and 
checked  out  by  the  instructor  24  hours  in  advance.  If  they  are  not  available 
to  the  instructor,  and  will  not  be  available  in  the  future,  the  Instructor  Guide 
should  be  revised.  Any  materials  that  are  inaccurate  should  be  eliminated 
from  the  Instructor  Guide  or  revised. 
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2.2.2  The  Instructor  Did  Not  Follow  the  Content  Guidelines  From  the  Lesson 
Plan  (Questions  83  and  84). 

Trainees  cannot  learn  everything  that  is  presented  to  them  during  a  course  of 
instruction,  so  they  need  help  in  determining  what  is  important  and  what  is 
not  important.  Not  all  statements  made  by  an  instructor  are  equally 
important.  Some  subtasks  or  steps  are  more  critical  than  others.  Instructors 
should  highlight  critical  subtasks  and  steps  for  the  trainees  by  (1) 
emphasizing  the  "need  to  know"  and  minimizing  the  "nice  to  know;"  (2) 
minimizing  extra  material  that  the  instructor  may  think  is  helpful  but  which 
is  actually  not  a  part  of  the  lesson;  and  (3)  refraining  from  telling  ancedotes 
not  directly  related  to  the  subtask  or  task  being  learned.  Instructors  should 
have  been  cautioned  about  this  during  the  Instructor  Training  Course.  This 
point  needs  to  be  repeated  often  and  instructors  should  be  observed 
periodically,  however,  because  projecting  one's  own  sense  of  what  is  important 
and  using  the  podium  to  establish  credentials  as  an  "expert"  are  normal. 
Instructors  will  engage  in  these  activities  without  being  aware  of  it.  Even 
very  highly  trained  instructors  need  to  be  cautioned  occasionally  about  this. 
Beware  of  the  argument  that  "stories"  motivate  trainees  and  relax  the 
training  atmosphere.  They  may  and  they  may  not.  In  any  event,  story 
telling  should  be  confined  to  breaks. 

The  argument  that  instruction  was  added  or  deleted  to  conform  to  current 
reality  should  be  pursued  with  the  instructor.  Sometimes  equipment  or 
doctrine  changes,  new  or  changed  publications  (TMs,  FMs,  etc.),  new  training 
guidance,  etc.,  will  have  forced  a  modification  to  the  lesson  plan  that, 
because  of  time  constraints,  has  not  been  made.  Instructors  may  be  aware  of 
these  changes  and  may  have  already  changed  the  actual  instruction.  If  this 
has  happened,  press  the  training  developer  to  make  the  changes  on  the  lesson 
plan  as  soon  as 'possible. 

It  may  also  be  the  case  that  the  instructor  did  not  reach  all  of  the  content 
in  the  lesson  materials  because  there  was  not  enough  time  in  the  schedule  to 
cover  all  the  material  in  the  lesson  plan.  The  time  scheduled  for  a  lesson 
should  allow  an  introdution,  lecture(s),  demonstration(s),  and  enough  practice 
trials  to  bring  almost  all  (at  least  90%)  trainees  to  standard  on  all  subtasks 
and  tasks.  This  should  all  be  accomplished  at  a  pace  that  the  trainees  can 
maintain.  If  there  are  no  obvious  inefficiencies,  enough  time  should  be 
allotted  for  these  training  events.  If  these  training  events  do  occur,  but 
training  time  is  not  efficiently  utilized,  these  inefficiencies  should  be  solved 
before  addressing  the  time  problem. 


2.3  TRAINING  ENVIRONMENT:  Number  of  Instructors 


i 

i 

THE  PROBLEM.  Instructional  personnel  were  not  present  in  sufficient  ! 

numbers  to  ensure  that  all  students  can  see,  hear,  and  receive  feedback.  For  ' 

example,  the  presentation  of  STATEMENTS  and  EXAMPLES  requires  every 
student  to  see  and  hear  what  is  being  presented.  For  PRACTICE  events,  | 

there  should  be  sufficient  instructors  or  assistant  instructors  to  provide  j 

necessary  FEEDBACK  and  FEEDBACK  HELP  to  every  student.  j 

EVIDENCE  for  an  insufficient  number  of  instructors  will  come  from  the 
Worksheets  Cl.l  Observation  of  Classroom  Instruction  and  C1.3  Observation 
of  Isolated  Practice 

Question  81.  Is  the  instructor/trainee  ratio  such  that  all  students  can 
see,  hear,  and  receive  FEEDBACK? 

RECOMMENDATIONS  for  training  program  modification. 

2.3.1  Not  Enough  Instructors  Present  for  Training  (Question  81). 

There  were  not  enough  instructors  available  to  insure  that  all  students  can 
see,  hear,  and  receive  feedback.  Enough  instructors/assistant  instructors  must 
be  assigned  to  training  so  that  the  training  can  be  carried  out  as  planned. 

This  must  include  a  sufficient  number  of  assistant  instructors  to  provide  the 
necessary  one-on-one  guidance  during  practice  activities. 

Errors  made  during  practice  become  learned  as  part  of  the  skill  if  not 
immediately  corrected.  They  then  have  to  be  "trained-out"  at  some  future 
training  event.  Assistant  instructors  must  be  on-hand  to  insure  that  every 
trainee  practices  the  task  until  he  reaches  the  training  standard,  correcting 
errors  in  performance  as  they  are  made.  Errors  pointed  out  during  critiques 
following  task  performance  will  not  correct  faulty  performance  unless 
immediately  followed  by  additional  practice  events  and  the  process  repeated 
until  error -free  performance  is  realized.  This  is  inefficient  —  it  is  far  better 
to  correct  errors  in  performance  as  they  occur.  If  providing  assistant 
instructors  in  the  required  numbers  is  not  feasible,  then  training  sessions 
should  be  lengthened  so  that  each  trainee  still  receives  the  one-on-one 
training  during  practice  activities.  Instead  of  two  instructors  training  four 
trainees  in  60  minutes  (30  minutes  per  trainee),  for  example,  one  instructor 
may  train  four  trainees  in  120  minutes  (30  minutes  per  trainee).  In  this  way 
each  trainee  still  receives  the  required  supervised  practice.  If  time  and 
number  of  instructors  cannot  be  changed,  then  training  objectives  must  be 
modified  to  reflect  what  can  be  trained  with  the  given  resources. 
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2 .4  TRAINING  ENVIRONMENT:  Training  Duration 


THE  PROBLEM.  The  time  devoted  to  this  training  was  either  so  short  that 
the  instruction  was  rushed  or  that  some  events  were  left  out,  or  so  long  that 
trainees  got  bored.  Either  extreme  is  bad,  with  short  time  allotments  harder 
to  compensate  for. 

EVIDENCE  for  too  much  or  too  little  training  time  will  come  from  the 
following  worksheets: 

0.1  Observation  of  Classroom  Instruction 
C1.2  Observation  of  Demonstrations 
C1.3  Observation  of  Isolated  Practice 
0.4  Observation  of  a  Performance  Test 
0.5  Observation  of  Written  Test 
0.6  Observation  of  an  Oral  Test 
0.7  Observation  of  Integrated  Practice 
0.8  Observation  of  Individual  Study 
Question  90.  Are  frequent  breaks  provided? 

Question  92.  Was  the  allotted  training  time  too  long  or  too  short? 

* 

RECOMMENDATIONS  for  training  program  modification. 

2.4.1  Insufficient  Breaks  or  Rest  Periods  (Question  90). 

Breaks  were  not  provided  or,  if  they  were,  they  were  too  short  or  infrequent. 
Rest  periods  of  at  least  5  to  10  minutes  should  be  allowed  after  each  hour  of 
instruction.  Longer  breaks  should  be  given  if  the  instruction  exceeds  3  or  4 
hours.  Insufficient  rest  periods  will  result  in  trainee  fatigue,  lack  of 
attention,  and  decreased  performance.  If  the  time  allotted  in  the  training 
schedule  does  not  allow  for  sufficient  breaks,  the  syllabus  should  be  adjusted 
accordingly  (see  recommendations  for  Question  92  below). 

2.4.2  Allotted  Time  Too  Short/Long  (Question  92). 

The  time  allotted  in  the  lesson  plan  (or  Program  of  Instruction  or  Training 
Schedule)  was  too  short/long  for  the  successful  accomplishment  of  the  training 
objectives.  Training  time  here  has  to  be  considered  along  with  training 
equipment  and  assistant  instructors.  The  lesson  plan  (or  Program  of 
Instruction  or  Training  Schedule)  should  specify  enough  training  equipment  and 
assistant  instructors  to  permit  each  trainee  to  practice  the  task  until  he  or 
she  gets  it  right  under  the  time  standard  specified  for  the  task.  The  problem 
is  one  of  planning.  You  will  have  to  find  out  how  many  training  sites  are 


scheduled,  how  many  trainees  are  in  a  usual  sized  class,  how  much  classroom 
(general)  instruction  time  is  provided,  how  much  time  of  instruction  is  planned 
for  each  individual  trainee,  and  the  time  standard  for  the  task.  From  this, 
you  can  determine  minimum  allotted  time.  An  example  follows. 

If  there  are  to  be  10  sites  (each  manned  by  an  assistant  instructor;  do  not 
count  any  site  that  is  not  so  manned)  with  five  trainees  at  each  site  (i.e.,  50 
trainees  for  10  sites  is  five  trainees  per  site),  30  minutes  of  general 
instruction,  10  minutes  of  individual  instruction  for  each  trainee,  where  each 
trainee  is  required  to  walk  through  the  task  one  time  and  then  practice  until 
he  or  she  can  perform  it  one  time  to  standard,  and  the  time  standard  for  the 
task  is  10  minutes,  then  the  total  time  that  must  be  allocated  is: 

General  instruction  =  30  minutes 

plus 

five  trainees  (per  site)  X  10  minutes  =  50  minutes 

(individual  instruction) 

plus 

two  repetitions  (at  least)  of  the  =  100  minutes 

task  X  10  minutes  (standard)  X 

five  trainees  (per  site)  _ 

180  minutes 


If  the  trainees  are  to  be  given  a  10-minute  break  every  hour,  then  an 
additional  20  minutes  will  be  needed  for  two  10-minute  breaks.  Note  that 
the  nearly  three  and  one-half  hours  does  not  allow  time  for  anyone  to  get  the 
task  wrong  during  practice  and  does  not  allow  any  buffer  time  for 
contingencies,  and  therefore  is  the  very  minimum  on  this  example.  Do  the 
arithmetic  for  the  task  to  determine  the  minimum  time  allotment.  If  you 
cannot  get  the  figures  you  need  for  input,  then  training  has  not  been 
carefully  planned.  This  problem  should  be  brought  to  the  attention  of  the 
training  developer  so  that  time  allocations  can  be  made.  If  you  do  the 
arithmetic  and  come  up  with  the  allotted  time  being  too  short  or  too  long, 
notify  the  training  developer  so  that  the  appropriate  adjustments  can  be 
made.  Do  not  shy  away  from  this  because  it  looks  complicated.  Insufficient 
time  is  a  typical  training  problem  that  will  occur  again  and  again.  If  you  do 
not  correct  it,  probably  no  one  will. 


2.5  TRAINING  ENVIRONMENT:  Attitudes  and  Motivation 


THE  PROBLEM.  Trainees  did  not  appear  interested  and  did  not  actively 
participate  in  the  instruction,  or  they  indicated  that  they  didn't  care  about 
the  training  or  didn't  want  to  learn.  They  may  even  have  displayed  hostility 
or  frustration  during  instruction. 

EVIDENCE  for  poor  trainee  attitudes  will  come  from  the  following 
worksheets: 

Cl.l  Observation  of  Classroom  Instruction 
C1.2  Observation  of  Demonstrations 
C1.3  Observation  of  Isolated  Practice 
C1.8  Observation  of  Individual  Study 
Cl.  10  Trainee  Reaction  Questionnaire 
Question  28.  Is  the  trainee  attitude  positive? 


RECOMMENDATIONS  for  training  program  modification. 

2.5.1  Poor  Trainee  Attitude  (Question  28). 

Trainees  generally  prefer  to  be  passive  learners.  In  many  ways  this  is 
reasonable  given  the  world  of  television,  the  passive  nature  of  public 
instruction  today,  and  the  use  of  so  many  films  and  slide  presentations  during 
military  training.  Training,  however,  cannot  be  a  passive  enterprise  for  the 
trainee.  Skills  are  developed  through  active  participation.  Instructors  should 
build  an  atmosphere  that  encourages  participation  by  being  candid,  friendly, 
non-critical,  and  eager  to  help  trainees  learn  and  master  tasks.  All  criticism 
should  be  positive  and  aimed  at  task  performance  rather  than  negative  or 
aimed  at  the  trainees.  It  is  the  task  performance  that  is  substandard,  not 
the  trainee.  The  instructor  should  also  make  a  concerted  effort  to  motivate 
the  trainees.  Ways  of  motivating  students  are  discussed  in  the 
recommendations  for  Question  27  in  the  section  "GENERAL  OBSERVATIONS: 
Instructor  Performance." 

There  are  a  number  of  other  causes  for  poor  trainee  attitude.  These  will  be 
identified  by  other  TEE  questions.  Some  examples  are  malfunctioning  training 
equipment  (Question  93),  uncomfortable  training  site  (Questions  85-88), 
material  too  difficult  for  student  reading  level  (Question  71),  trick  test 
questions  (Question  16),  and  so  forth.  When  the  recommendations  for  solving 
each  of  these  training  problems  are  implemented,  trainee  attitude  should 
improve. 


THE  PROBLEM.  Instructors  did  not  demonstrate  a  good  grasp  of  how  to 
conduct  effective  training.  Since  the  instructor  is  usually  the  medium  through 
which  the  information  and  guidance  flows  to  the  trainee,  attention  is  rooted 
on  the  instructor.  Flaws  in  presentation  technique  stand  out  and  can  destroy 
the  positive  training  benefits  of  well  designed  training.  All  instructors  should 
have  attended  an  Instructor  Training  Course.  Any  who  have  not  should  be 
scheduled  to  attend  such  a  course  as  soon  as  possible.  Subject  matter 
expertise  will  not  compensate  for  a  lack  of  skill  in  training  delivery. 

EVIDENCE  for  substandard  instructor  performance  will  come  from  the 
following  worksheets: 

Cl.l  Observation  of  Classroom  Instruction 
C1.2  Observation  of  Demonstrations 
Cl. 3  Observation  of  Isolated  Practice 
C1.4  Observation  of  a  Performance  Test 
Cl. 5  Observation  of  a  Written  Test 
C1.6  Observation  of  an  Oral  Test 
C1.7  Observation  of  Integrated  Practice 

Question  27.  Are  attention-getting/motivational  techniques  employed? 

Question  89.  Is  the  instructor's  attitude  positive? 

* 

RECOMMENDATIONS  for  training  program  modification. 

2.5.2  Attention  Getting  or  Motivational  Techniques  Not  Employed  (Question 

277 

No  attempt  was  made  to  motivate  the  trainees.  Trainee  motivation  must  be 
a  central  consideration  during  the  design  of  any  training  program  if  it  is  to 
be  successful.  Training  that  does  not  capture  the  enthusiasm  of  the  student 
will  be  less  than  optimally  effective.  Some  ways  to  motivate  students  are  to 
tell  them  why  the  training  is  necessary,  how  the  task  fits  in  with  their 
eventual  job  duties,  and  what  would  happen  to  them  if  they  did  or  did  not 
learn  the  task. 


The  purpose  for  the  training  should  be  given  to  the  soldiers.  The  training 
objective  tells  the  trainee  what  he  or  she  will  be  learning  to  do.  The 


purpose  tells  the  trainee  why  he  or  she  has  to  learn  it  and  why  it  must  be 
learned  now.  Most  training  developers  use  this  opportunity  to  motivate  the 
trainees.  For  example,  if  the  training  objective  is  to  learn  to  fix  "X,"  the 
purpose  will  tell  them  why  they  (instead  of  others)  have  to  learn  to  fix  "X" 
and  why  they  have  to  learn  to  fix  it  now  instead  of  some  other  time.  Such 
trainee  put-offs  as,  "You  need  it  to  pass  the  test,"  or  "You  need  it  to  get 
the  MOS,"  or  "You  need  it  on  the  job"  should  be  avoided.  If  there  is  a  good 
reason  for  giving  the  training  in  the  course,  then  the  reason  should  be  given 
to  the  trainees.  If  there  is  no  good  reason,  then  perhaps  the  training  should 
not  be  given  in  the  course. 

The  trainees  should  also  be  told  how  each  unit  of  training  fits  in  with 
eventual  job  duties.  Tying  training  to  eventual  job  duties  helps  soldiers 
develop  a  feel  for  the  whole  job,  giving  them  a  framework  for  hooking  the 
various  tasks  together  in  a  sensible  fashion. 

Finally,  trainees  should  be  told  what  would  happen  to  them  if  they  did  or  did 
not  learn  to  perform  the  required  tasks.  If  there  are  no  rewards  for  learning 
the  required  tasks,  and/or  there  are  no  penalties  for  not  learning  them,  it  will 
be  very  hard  to  motivate  the  trainees.  In  essence,  it  will  make  no  difference 
to  them  if  they  learn  or  not.  Consequences  for  learning/not  learning  may  be 
at  the  course  level  rather  than  the  task  level  but  there  should  be 
consequences.  All  rewards  and  penalties  should  be  made  known  to  the 
trainees. 

2-5.3  Negative  Instructor  Attitude  (Question  89). 

Instructors  should  exhibit  a  positive  attitude  at  all  times  toward  the  training, 
themselves,  and  the  trainees.  Instructors  should  not  use  the  podium  to  launch 
their  personal  beliefs  about  the  state  of  the  Army,  the  quality  of  today's 
soldiers,  pay  and  allowances,  readiness,  combat  capabilities  of  units  and  items 
of  equipment,  instructional  objectives/sequence/organization,  training  materials, 
job  aids,  or  anything  else.  The  instructor  is  the  model  that  the  trainees  are 
expected  to  follow.  This  model  should  be  positive.  Instructors  who  do  not 
display  a  positive  attitude  should  be  cautioned  about  this  by  the  training 
manager.  Continued  demonstration  of  a  poor  attitude  by  an  instructor,  no 
matter  how  friendly,  entertaining,  or  technically  proficient,  should  be  cause 
for  seriously  considering  his  removal  from  instructor  duties. 


2.6  TRAINING  ENVIRONMENT:  Training  Equipment 

THE  PROBLEM.  Training  devices  (including  actual  equipment  used  during  the 
training)  did  not  work  properly,  resulting  in  degradation  of  the  learning 
process. 

Cl.l  Observation  of  Classroom  Instruction 


C1.3  Observation  of  Isolated  Practice 


C1.6  Observation  of  an  Oral  Test 


C1.8  Observation  of  Individual  Study 

Question  93.  Does  the  training  device/equipment  used  in  training 
function  properly? 

RECOMMENDATIONS  for  training  program  modif icaton. 

2-6.1  Non-Functioning  or  Malfunctioning  Training  Equipment  (Question  93). 

The  training  equipment  did  not  work  properly.  It  is  almost  impossible  to 
train  students  to  standard  using  equipment  and  materials  that  do  not  work 
properly.  For  example,  in  a  task  involving  meter  calibration,  if  the 
equipment  is  not  operating  as  it  should,  students  will  not  be  able  to  see  the 
task  demonstrated  properly  or  practice  the  task  under  normal  conditions.  In 
adddition,  malfunctioning  training  equipment  undermines  trainee  confidence  in 
the  actual  equipment,  and  causes  students  to  question  the  competence  of  the 
instructors.  Training  equipment  should  be  secured  by  the  instructors  well  in 
advance  of  the  training  and  checked  to  insure  that  it  works  properly. 


2.7  TRAINING  ENVIRONMENT:  Distractions 


THE  PROBLEM.  The  training  was  interrupted,  or  the  trainees  were  distracted 
from  training,  by  such  things  as  equipment  failure,  instructor  absence,  VIP 
orientations,  high  ranking  observers  at  the  training  site,  administrative 
requirements,  etc.  This  may  look  like  a  non-training  problem  but  the  effects 
of  such  interruptions  are  on  trainee  skill  and  knowledge.  Test  data  will 
indicate  where  trainee  performance  deficiencies  are.  Remedial  training  may 
need  to  be  scheduled  to  bring  trainee  performance  levels  up  to  standard. 

EVIDENCE  for  the  presence  of  distractions  during  training  will  come  from  the 
following  worksheets: 

Cl.l  Observation  of  Classroom  Instruction 

C1.2  Observation  of  Demonstrations 

C1.3  Observation  of  Isolated  Practice 

C1.4  Observation  of  a  Performance  Test 

C1.5  Observation  of  a  Written  Test 

C1.6  Observation  of  an  Oral  Test 

C1.7  Observation  of  Integrated  Practice 

C1.8  Observation  of  Individual  Study 

Question  86.  Is  the  instruction  free  of  distractions? 

RECOMMENDATIONS  for  training  program  modification. 

The  training  session  was  seriously  distracted  at  one  or  more  points  by  such 
things  as  interrupted  training  (VIP  coming  into  teaching  area,  training 
movement  for  administrative  purposes,  power  failure,  training  aid  malfunction, 
etc.),  excessive  noise  over  which  the  instructor  has  no  control,  sudden 
instructor  or  trainee  illness,  etc.,  and/or  factors  in  the  training  environment 
were  severe  enough  to  have  adversely  affected  trainee  attention  and  learning. 
The  specific  problem  in  this  case  will  be  supplied  by  the  analyst.  There  is 
very  little  that  can  be  done  to  prevent  training  distractions.  When  they 
occur,  however,  soldier  learning  usually  suffers.  The  problem  in  this  case  is 
what  to  do  about  that.  Test  data  will  indicate  where  soldier  performance 
deficiencies  are.  Remedial  training  will  need  to  be  scheduled  to  bring  soldier 
performance  levels  up  to  standard. 

There  are  two  special  cases  of  training  distractions  which  merit  specal 
attention: 

2.7.1  Obsjrvers  Distracting. 

Observers  at  the  training  site  distracted  the  soldiers  from  training.  The 
instructional  staff  usually  has  little  control  over  the  number,  rank,  and 
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activities  of  observers.  If  the  observers  were  so  distracting  that  they 
actually  interrupted  training,  a  remedial  training  session  may  have  to  be 
scheduled.  If  they  were  just  bothersome,  steps  may  need  to  be  taken  to 
actively  close  the  training  site  to  anyone  who  does  not  belong  there.  Many 
times  observers  are  just  "official  looking,"  and  actually  have  no  real  business 
in  the  training  site. 

2.7.2  Training  Interrupted. 


Training  was  interrupted  or  stopped  for  some  reason.  You  may  have  to 
question  the  training  observer,  soldiers,  and/or  instructors  to  find  out  what 
happened.  If  the  interruption  was  serious,  remedial  training  may  have  to  be 
scheduled.  Serious  interruptions  are  those  that  result  in  subtasks  or  tasks  not 
being  completely  covered  or  practiced  to  standard  by  everyone.  Acts  of  God 
and  higher  headquarters  cannot  be  put  off.  Nuisance  interruptions,  however, 
can  often  be  controlled  with  proper  planning. 


3. 


PRESENTATION 


3.1  PRESENTATIONS  Prerequisites 

THE  PROBLEM.  The  instructor  did  not  review  course  entry  skills  or  he 
neglected  to  verify  mastery  of  prerequisite  skills  prior  to  new  instruction. 

EVIDENCE  for  inadequate  assessment  of  prerequisite  skills  and  knowledge  will 
come  from  the  following  worksheets: 

Cl.l  Observation  of  Classroom  Instruction 

C1.2  Observation  of  Demonstrations 

C1.3  Observation  of  Isolated  Practice 

C1.7  Observation  of  Integrated  Practice 

Question  29.  Are  course  ENTRY  SKILLS  reviewed? 

Question  30.  Is  mastery  of  prerequisite  skills  verified  prior  to  new 
instruction? 

RECOMMENDATIONS  for  training  program  modification. 

3.1.1  Prerequisite  Knowledge  Not  Assessed  (Questions  29  and  30). 

The  instructor  did  not  question  the  trainees  about  what  they  were  already 
supposed  to  know.  Sometimes  (but  rarely)  this  is  not  necessary  because  the 
trainees  have  befen  pretested  on  baseline  knowledge  and  skills  and  the  results 
provided  to  the  instructor.  If  the  trainees  have  not  been  pretested,  however, 
the  instructor  should  ask  enough  questions  to  ascertain  that  the  trainees  do 
indeed  know  what  the  lesson  assumes  they  know.  If  they  do  not,  remedial 
training  is  needed  before  task  training  begins.  If  this  group  of  soldiers  is 
typical  of  future  input,  training  has  to  be  modified  to  include  this  additional 
training. 
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3.2  PRESENT/*  HON:  Training  Objectives 


THE  PROBLEM.  Trainees  were  not  mentally  prepared  to  receive  the  training 
because  training  objectives  were  not  presented  first.  This  should  be  required 
by  the  lesson  plan  and  should  occur  at  the  start  of  each  lesson. 

EVIDENCE  for  the  fact  that  the  training  objectives  were  not  clearly 
communicated  to  the  trainees  will  come  from  the  following  worksheets: 

Cl.l  Observation  of  Classroom  Instruction 

Cl. 2  Observation  of  Demonstrations 

C1.3  Observation  of  Isolated  Practice 

CI.7  Observation  of  Integrated  Practice 

Question  31.  Are  OBJECTIVES  presented  to  the  student? 

RECOMMENDATIONS  for  training  program  modification. 

3.2.1  Training  Objectives  Not  Presented  to  Students  (Question  31). 

The  trainees  were  not  given  the  training  objectives  in  a  clear  and  easily 
understandable  format.  All  training  objectives  have  three  parts  that  must  be 
given  to  the  trainee:  (1)  the  task  statement  that  tells  the  trainees  what  they 
must  be  able  to  do  following  training;  (2)  the  conditions  statement  that  tells 
them  what  they  will  have  to  do  it  with  and  the  conditions  under  which  they 
will  have  to  perform;  and  (3)  the  standard  that  tells  the  trainees  how 
accurately  and/dr  quickly  they  will  have  to  do  it  to  get  a  GO.  All  three 
parts  must  be  in  the  lesson  plan  and  all  three  parts  must  be  given  to  the 
trainees  at  the  beginning  of  training. 
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3.3  PRESENTATION:  Course  Administration  Directions 

THE  PROBLEM.  Course  administration  directions,  contained  in  the  lesson 
plans,  are  incomplete  or  make  unrealistic  demands  of  students  and  instructors. 
If  the  lesson  plans  do  not  completely  outline  how  the  course  and  individual 
lessons  are  to  be  administered,  then  there  is  no  guarantee  that  the  training 
program  will  achieve  the  training  objectives.  Inadequate  or  unrealistic  course 
administration  directions  will  cause  instructors  to  abandon  the  lesson  plan  and 
give  instructor -specific  training. 

EVIDENCE  for  inadequate  course  administration  directions  will  come  from  the 
following  worksheets: 

Cl.l  Observation  of  Classroom  Instruction 
C1.2  Observation  of  Demonstrations 
C1.3  Observation  of  Isolated  Practice 
C1.7  Observation  of  Integrated  Practice 
C1.8  Observation  of  Individual  Study 

Question  79.  Are  course  administration  directions  complete? 

Question  80.  Do  course  administration  directions  make  realistic  demands 
of  students  and  instructors? 


RECOMMENDATIONS  for  training  program  modification. 


Course  administration  directions  were  incomplete  or  nonexistent  and 
instructors  were  left  to  their  own  judgment  as  to  how  to  conduct  training.  A 
complete  lesson  plan  should  do  the  following: 

a)  Call  for  the  dissemination  of  the  required  enabling  knowledge  in 
either  a  lecture  or  a  hand-out. 

b)  Call  for  a  demonstration  of  the  task  (in  its  entirety). 

c)  Require  demonstrations  for  each  subtask  in  the  lesson  and  on 
integrated  demonstration  after  subtask  training. 

d)  Call  for  supervised  hands-on  practice. 

e)  Require  that  each  subtask  and  task  be  practiced. 

f)  Specify  that  practice  on  each  subtask/task  will  be  performed  by 
each  trainee  to  a  specified  standard  (or  to  the  standard  specified 
in  the  training  objective). 


g)  Provide  guidance  for  assistant  instructors  (AIs)  by  informing  them 
what  to  look  for  during  practice  and  how  to  correct  faulty 
performance. 

Each  of  these  features  is  described  in  detail  below: 

a)  Does  the  lesson  plan  call  for  the  dissemination  of  the  required 
enabling  knowledge  in  either  a  lecture  or  a  hand-out? 

Terms,  concepts,  or  other  subject  matter  that  are  included  in  a 
lesson  to  enhance  the  trainee's  understanding  of  the  tasks  to  be 
learned  is  referred  to  as  "enabling  knowledge."  The  lesson  plan 
should  specify  the  enabling  knowledge  needed  by  the  trainees  for 
task  performance.  It  should  specify,  as  a  first  event  in  any 
sequence  of  training  events,  a  discussion  or  lecture  where  the 
instructor  establishes  the  terms  and  concepts  needed  by  him  or 
her  and  the  trainees  to  talk  about  the  task. 


The  lesson  plan  should  specify  what  enabling  knowledge  is  to  be 
presented  and  prescribe  procedures  (a  set  of  rules,  a  formula  to 
follow,  etc.)  for  presenting  it  since  the  trainees  must  readily 
understand  the  material  and  be  able  to  converse  easily  about  the 
task  using  corrective  terms  and  concepts.  The  lesson  plan  should 
specify  that  training  not  progress  to  a  practice  event  until  the 
instructor  is  sure  that  the  trainees  understand  and  can  use  the 
appropriate  terms  and  concepts. 

The  lesson  plan  should  specify  the  required  terms  and  concepts. 
These  terms  and  concepts  include  such  things  as  names  or  labels 
for'  things,  introduction  of  new  equipment,  explanations  of 
working  relationships  between  parts,  the  establishment  of  required 
readings  on  gauges  and  dials,  the  introduction  of  rules  or 
strategies,  the  explanation  of  underlying  concepts,  etc. 

b)  Does  the  lesson  plan  call  for  a  demonstration  of  the  task  in  its 
entirety? 

The  lesson  plan  should  call  for  a  demonstration  of  each  task,  in¬ 
cluding  the  appropriate  steps  or  subtasks.  The  lesson  plan  should 
provide  guidance  to  the  instructor  in  conducting  the  demonstra¬ 
tion;  it  should  prescribe  how  the  demonstration  is  to  be  con¬ 
ducted.  Merely  listing  the  task  to  be  demonstrated,  and  direct¬ 
ing  the  instructor  to  demonstrate  the  necessary  maintenance, 
assembly  disassembly,  or  operating  procedures  is  not  sufficient. 

Typically  the  lesson  plan  should  call  for  demonstrations  to  be 
conducted  by  the  instructor  on  the  actual  equipment  (or  a  high 
fidelity  mock-up)  in  full  view  of  the  trainees  being  trained.  This 
provides  the  trainees  with  a  live  model  of  the  skilled  per¬ 
formance  required,  and  allows  the  trainees  to  stop  the  instructor 
to  ask  questions.  Films,  slides,  or  other  pictorial  methods  are 
usually  less  effective  methods  of  demonstrating  tasks,  and 


should  only  be  used  when  time  and  resources  do  not  permit  a 
live  demonstration  by  the  instructor. 


In  most  cases  the  lesson  plan  should  call  for  a  formal 
demonstration  rather  than  a  walk-through  or  talk-through.  A 
walk-through  or  talk-through  may  be  adequate  for  demonstrating 
relatively  simple  tasks  where  skilled  performance  is  not  required, 
but  it  is  a  poor  substitute  for  a  formed  demonstration  in  most 
applications.  Extensive  supervised  practice  following  a  walk¬ 
through  or  talkthrOugh  can  sometimes  compensate  for  the  lack  of 
a  formal  demonstration,  but  in  general  the  lesson  plan  should 
provide  for  a  formal  demonstration. 

If  there  are  subtasks  in  the  lesson,  are  demonstrations  required 
for  each  subtask  and  is  a  combined  demonstration  required  after 
subtask  training? 

A  subtask  is  a  part  of  a  larger  task  that  can  be  performed 
independent  of  the  larger  task.  If  there  are  subtasks  in  the 
lesson,  the  lesson  plan  should  call  for  the  subtasks  to  be 
demonstrated  either  as  part  of  the  overall  task  or  as  separate 
subtasks.  When  a  task  consists  of  a  number  of  related  subtasks 
that  are  part  of  a  larger  overall  task,  the  lesson  plan  should  call 
for  a  demonstration  of  each  followed  by  a  demonstration  of  the 
full  task  to  include  each  of  the  subtasks  performed  sequentially. 
The  lesson  plan  should  specify  the  demonstration  sequence  in 
considerable  detail  and  show  how  to  schedule  demonstrations  and 
practice  together. 

Does  the  lesson  plan  call  for  supervised  hands-on  practice? 

The  lesson  plan  should  call  for  practice  of  the  task  where 
practice  is  defined  as  hands-on  performance  of  the  task,  on  the 
equipment  listed  in  the  training  objective,  supervised  by  an 
instructor  or  assistant  instructor,  during  training.  Practice 
events  are  essential  to  skill  development.  No  matter  how  good 
lectures,  demonstrations,  "talk-throughs,"  or  "walk-throughs"  are, 
trainees  will  not  learn  to  perform  tasks  unless  they  practice 
them. 


Does  the  lesson 
practiced? 


>lan  require  that  each  subtask  and  task  be 


Both  subtask  and  task  practice  should  be  specified  in  the  lesson 
plan.  Subtask  practice  is  practice  on  a  part-task,  or  on  a  step 
in  a  long  sequence,  or  on  a  particular  set  of  actions  that  are 
part  of  a  task.  Subtask  practice  is  necessary  when  a  task  is 
composed  of  many  subtasks  (where  the  sequence  of  actions  is  too 
long  and  involved  for  the  trainee  to  master  in  one  practice 
session).  It  is  more  efficient  to  have  the  trainee  master  the 
sequence  by  groups  of  subtasks. 
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After  several  sub  tasks  have  been  performed  to  standard,  then  the 
trainees  can  begin  to  put  them  together.  The  practice  event 
where  the  trainee  attempts  to  put  the  subtasks  together,  to 
practice  the  entire  task  under  the  training  condition  to  the 
training  standard,  is  called  whole-task  practice. 

f)  Does  the  lesson  plan  specify  that  practice  on  each  subtask/task 
will  be  performed  by  each  trainee  to  a  specified  standard  (or~to 
the  standard  specified  in  the  training  objective)? 

The  lesson  plan  should  require  each  trainee  to  practice  each  task 
and  subtask  until  he  or  she  meets  the  standard  listed  in  the 
training  objective  at  least  once.  No  trainee  should  be  allowed  to 
progress  in  training  without  having  satisfied  the  training 
objective.  Novel  or  difficult  tasks  may  require  more  than  one 
demonstration  of  proficiency  by  each  trainee. 

g)  Does  the  lesson  plan  provide  guidance  for  assistant  instructors 
(AIs)  that  tells  them  what  to  look  for  during  practice  and  how  to 
correct  faulty  performance? 

Progress  should  be  made  during  practice.  As  obvious  as  this 
point  seems,  it  is  often  ignored.  Progress  is  crucial,  for  if  it 
does  not  occur,  the  trainee  is  practicing  at  least  some  incorrect 
performance.  Furthermore,  because  progress  should  occur,  failure 
to  make  progress  is  a  good  indication  that  something  in  the 
training  needs  correcting.  Judgments  as  to  whether  progress  is 
occurring  must  be  made  by  AIs  who  are  monitoring  practice. 

This  means  that  the  lesson  plan  should  provide  guidance  for  the 
AIs'  that  tells  them  what  performance  problems  are  likely,  how 
to  identify  them,  how  to  provide  feedback,  and  what  remedial 
exercises  to  use  to  correct  faulty  performance.  For  example, 
the  lesson  plan  should  specify  those  aspects  of  the  tasks  that  are 
likely  to  cause  problems  for  the  trainees,  list  common  trainee 
errors  on  the  tasks  being  practiced,  and  direct  the  instructor  to 
make  feedback  regarding  trainee  errors  immediately,  specific  to 
the  actions  performed,  and  free  of  ridicule. 

3.3.2  Course  Administration  Directions  Unrealistic  (Question  80). 

Course  administration  directions  made  unrealistic  demands  on  students  or 
instructors.  Students  and  instructors  are,  after  all,  only  human.  Course 
administration  directions  should  take  into  account  the  qualifications  and 
limitations  of  students  and  instructors  and  should  not  require  them  to  do  what 
is  beyond  their  capability.  The  "fix"  required  depends  upon  the  specific 
unrealistic  demand.  For  example,  an  instructor  cannot  be  in  more  than  one 
place  at  the  same  time.  If  the  course  directions  require  one  instructor  to 
supervise  hands-on  practice  for  30  students  in  a  30  minute  time  period,  the 
directions  are  clearly  unrealistic.  In  this  case,  you  would  recommend  that  a 
sufficient  number  of  AIs  be  provided  or  that  time  allotted  for  practice  be 
increased. 
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3.4  PRESENTATION:  Adequacy 


THE  PROBLEM.  The  basic  presentation  components  are  inadequate.  Required 
presentation  components  are  missing;  statements  for  concepts,  procedures,  or 
rules  are  incomplete  or  inadequate;  or  statement  help  provides  insufficient 
explanation.  (Adequacy  criteria  for  examples,  practice  and  feedback  can  be 
found  in  the  sections  on  Examples,  Practice,  and  Feedback,  respectively.) 

EVIDENCE  for  inadequacy  of  presentation  components  will  come  from  Work 
Sheet  Cl.l  Observation  of  Classroom  Instruction. 

Question  32.  Are  basic  PRESENTATION  COMPONENTS  present? 

Question  33.  Are  STATEMENTS  complete? 

Question  34.  Are  STATEMENTS  for  CONCEPTS,  PROCEDURES,  or 
RULES  adequate? 

Question  35.  Does  STATEMENT  HELP  provide  sufficient  explanation? 
RECOM MEND ATIONS  for  training  program  modification. 

3.4.1  Instructional  Presentation  Incomplete  (Question  32). 

The  instructional  presentation  lacks  one  or  more  of  the  required  presenta.tion 
components.  The  term  "instructional  presentation"  covers  all  the  ways  that 
instruction  can  be  presented.  This  indudes  printed  self-study  materials, 
lectures,  computer-assisted  instruction,  films  or  videotapes,  tape-slide 
presentations,  audio  tape,  videodisc,  one-on-one  tutorials,  and  any  combination 
of  these. 


For  a  presentation  to  be  complete,  it  must  contain  certain  components, 
depending  on  the  task/content  level.  There  are  four  main  presentation 
components: 


STATEMENT  The  student  is  given  a  statement  of  a  fact,  a 

concept  definition,  the  steps  of  a  procedure  or  rule, 
or  a  statement  of  a  principle. 

EXAMPLE  The  student  is  told  or  shown  how  a  statement  of  a 

concept,  procedure,  rule,  or  principle  applies  in  a 
specific  case. 


PRACTICE  The  student  is  asked  to  supply  part  or  all  of  a  fact 

REMEMBERING  statement,  concept  definition,  the  steps  of  a 

procedure  or  rule,  or  a  statement  of  a  principle. 
The  student  is  also  given  feedback  about  the 
correctness  of  his  answer. 


PRACTICE 

USING 


The  student  is  asked  to  use  a  concept  definition, 
procedure,  rule,  or  principle  on  a  specific  case  to 
which  it  applies,  and  is  given  feedback  about  the 
quality  of  his  performance. 


S  ■  -•*.  '  w  ^  J 
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Different  combinations  of  these  components  are  required,  depending  on  the 
task  level  of  the  objective.  Figure  1  shows  which  presentation  components 
are  required  for  each  task  level.  In  order  to  use  Figure  1,  check  the 
presentation  component  used  and  the  task  level  of  the  instructional 
presentation.  The  table  will  indicate  whether  or  not  the  presentation 
component  is  required.  There  is  only  one  problem  area,  the  USE-UNAIDED 
objective  task  level.  This  will  require  a  STATEMENT  and  PRACTICE 
REMEMBERING  (with  feedb  zk)  only  if  the  REMEMBER  objective  was  not 
taught  recently,  or  if  the  students  do  not  fall  into  the  indicated  category. 


3.4.2  Statements  Incomplete  ( 


ion  33). 


The  fact,  concept,  procedure,  rule,  or  principle  statement  was  incomplete.  A 
statement  cannot  be  considered  as  present  in  the  instruction  unless  it  is 
complete.  Statements  should  have  the  following  parts: 


o  FACTS: 


The  complete  fact  or  set  of  facts  should  be 
presented. 


o  CONCEPTS:  All  critical  characteristics  and  how  they  are 

combined  should  be  given. 

o  PROCEDURES:  All  steps  should  be  given  in  the  correct  order. 


o  RULES: 


o  PRINCIPLES: 


3.4.3  Statements  Inac 


All  steps  and  branching  decisions  should  be  given  in 
the  correct  order. 

All  potential  component  concepts,  procedures,  rules, 
and/or  principles  should  be  given  (unless  remember¬ 
ing  them  can  be  considered  an  entry  skill)  along 
with  the  principle  itself  (a  general  rule  giving 
typically  incomplete  guidance  on  how  to  construct  a 
higher  order  rule  from  the  component  parts). 

ate  (Question  34). 


The  concept,  procedure  or  rule  statement  was  not  clearly  stated.  For 
difficult  concept  tasks,  it  is  often  desirable  to  include  not  only  the  concept 
characteristics,  but  also  a  decision  strategy  or  rule  for  making  classifications. 
For  procedures,  each  step  should  have  only  one  action.  If  there  is  more  than 
one,  the  step  should  be  broken  up.  Also,  explicit  safety  precautions  should  be 
included  as  separate  steps.  For  rules,  if  a  formula  is  used,  symbols  must  be 
defined. 


3.4.4  Statement  Help  Provides  Insufficient 


lanation  (C 


tion  33). 


The  statement  help  did  not  aid  the  student  in  better  understanding  and 
remembering  the  statement.  Methods  of  providing  help  indude: 
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1.  Giving  a  mnemonic  (memory  trick). 

2.  Giving  a  general  example  of  how  the  statement  can  be  used. 

3.  Explaining  why  the  statement  is  important. 

4.  Explaining  how  it  came  about,  how  it  fits  in  the  course,  or  how 
it  relates  to  something  the  student  already  knows. 

5.  Explaining  the  terms  in  the  statement. 

6.  Representing  the  statement  with  pictures,  symbols,  flowcharts, 
tables,  etc. 

More  detailed  guidelines  for  help  for  various  types  of  content  are  given 
below. 

HELP  for  REMEMBER-FACT  objectives  should: 


1.  Make  the  facts  easier  to  learn  by  using  one  or  more  of  these 
techniques: 

a.  Relate  the  new  material  to  previous  learning. 

b.  Illustrate  the  material  in  a  memorable  way  by  using 
graphics. 

c.  Use  a  rhyme,  acronym,  chunking,  or  other  memory  aid. 

2.  Be'dearly  linked  to  the  information  to  be  learned. 

3.  Not  add  substantially  to  the  memory  load. 

4.  Include  only  essential  information. 

HELP  for  REMEMBER-RULE,  -CONCEPT,  or  -PROCEDURE  objectives  should: 


1.  Make  the  steps,  operations  or  critical  characteristics  easier  to 
remember  by  using  one  or  more  of  these  techniques: 

a.  Present  a  verbal  menory  aid,  such  as  an  acronym  or  jingle. 


Organize  the  steps,  operations,  or  critical  characteristics 
into  small  groups  (chunks). 

Illustrate  each  step,  operation,  or  critical  characteristic  in 
a  memorable  way  by  use  of  graphics. 

Present  a  memorable  description  or  example  of  the  steps, 
operations,  or  characteristic. 
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e. 


Relate  the  new  material  to  previous  knowledge. 


2.  Be  clearly  linked  to  the  material  to  be  learned. 

3.  Not  add  substantially  to  the  memory  load. 

4.  Include  only  essential  information. 

HELP  for  USE-CONCEPT  objectives  should: 

1.  Make  the  concept  easier  to  use  by  one  or  more  of  the  following 
techniques: 

a.  Relate  to  what  the  learner  already  knows. 

b.  Highlight  some  or  all  of  the  critical  characteristics  of 
typical  example(s). 

c.  Redefine  or  explain  terms  which  were  previously 
introduced. 

d.  Explain,  expand,  or  restate  critical  characteristics. 

e.  Present  a  search  strategy  for  recognizing  examples. 

2.  Relate  directly  to  the  generality. 

HELP  for  USE-PROCEDURE  objectives  should: 

1.  Make  the  procedure  easier  to  use  and  overcome  expected 

problems  or  common  mistakes  by  one  or  more  of  the  following 
techniques: 

a.  Relate  all  or  part  of  the  procedure  to  the  learner's  prior 
general  knowledge  or  previous  learning  of  specific  steps. 

b.  Give  the  reasoning  behind  all  or  part  of  the  procedure. 

c.  Explain  common  errors,  expand,  or  restate  each  of  the 
steps  as  needed. 

d.  Redefine  or  explain  terms  which  were  previously 
introduced  but  may  need  to  be  reviewed. 

e.  Use  graphics  or  illustrations  for  emphasis  or  clarification. 


2.  Be  clearly  related  to  the  procedure  statement  in  form  and 
content. 

HELP  for  USE-RULE  objectives  should: 

1.  Make  the  rule  easier  to  use  and  overcome  expected  problems  or 
common  mistakes,  by  doing  one  or  more  of  the  following: 
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a.  Present  an  application  of  the  complete  rule  to  a  given 
situation. 

b.  Relate  all  or  part  of  the  rule  to  the  learner's  prior 
general  knowledge  or  previous  learning. 

c.  Expand,  restate,  or  explain  difficult  steps  or  operations  in 
greater  detail. 

d.  Point  out  and  explain  common  errors. 

e.  Redefine  or  explain  terms  which  were  previously 
introduced  but  may  need  revision. 

f.  Give  the  reasoning  behind  all  or  part  of  the  rule. 

g.  Use  graphics  or  illustrations. 

h.  Present  alternate  representation  of  the  rule. 

2.  Be  clearly  related  to  the  generality. 
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3.5  PRESENTATION:  Sequencing 

THE  PROBLEM.  Training  did  not  progress  as  it  should  have.  Objectives 
within  a  lesson  were  not  sequenced  properly,  or  the  lessons  themselves  were 
not  sequenced  properly  within  the  course. 

EVIDENCE  for  an  improper  training  progression  will  come  from  the  following 
worksheets: 

C1.2  Observation  of  Demonstrations 
C1.3  Observation  of  Isolated  Practice 

C1.9  Rating  of  Process  Evaluation  Questions  Covering  a  Series  of 
Training  Events 

Question  75.  Are  the  OBJECTIVES  (TLOs  and  LOs)  within  each  LESSON 
sequenced  properly9 

Question  76.  Are  the  LESSONS  sequenced  properly  within  the  course? 

3.5.1  Objectives  (TLOs  and  LOs)  Within  a  Lesson  Improperly  Sequenced 
(Question  75).  ~ 

The  sequence  of  objectives  within  a  lesson  did  not  follow  the  natural  order  of 
learning  inherent  in  those  objectives.  The  purpose  of  proper  sequencing  of 
learning  objectives  is  to  assure  that  each  learning  objective  is  placed  in  the 
best  relationship  with  other  learning  objectives  to: 

o  prdduce  the  most  learning  in  the  shortest  period  of  time 

o  help  the  trainee  make  the  transition  from  one  skill  or  body  of 

knowledge  to  another 

o  assure  that  the  supporting  knowledge  and  skills  are  acquired 
before  dependent  subject  matter  is  introduced. 

Keep  in  mind  that  there  are  no  hard  rules  for  sequencing  objectives. 

However,  there  are  rules  of  thumb  which  will  help  to  devise  a  reasonable 
sequence: 

In  order  to  sequence  two  learning  objectives,  you  must  first  determine  the 
relationship  between  them.  Two  learning  objectives  may  have: 

1)  a  dependent  relationship,  in  that  mastery  of  one  requires  prior 
mastery  of  the  other. 

2)  an  independent  relationship,  in  that  they  are  totally  unrelated 
and  independent  of  each  other;  or 

3)  a  supportive  relationship,  in  that  some  transfer  of  learning  takes 
place  from  one  learning  objective  to  the  other. 
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The  table  below  contrasts  the  three  types  of  dependent,  independent  and 
supportive  relationships.  The  table  or  examples  of  each  of  these  relationships 
and  shows  how  the  relationship  affects  sequencing. 

Some  effects  of  sequencing  are  described  below: 

1.  Sequencing  effects  are  long-range.  The  advantages  or 
disadvantage  of  using  any  sequencing  scheme  will  not  likely  show 
up  immediately.  Therefore,  end-of-course  tests  should  be  used 
for  evaluating  effectiveness  of  sequencing  techniques.  Within- 
course  tests  of  small  portions  of  the  course  are  not  likely  to 
reveal  the  true  effects  of  sequence. 

2.  Sequence  is  important  to  low-aptitude  students.  Students  who 
have  a  high  aptitude  for  the  subject  matter  will  learn  it  in  spite 
of  sequencing.  The  lower  aptitude  of  the  learner  for  the 
content,  the  more  important  it  becomes  that  some  type  of 
sequence  and  structure  is  provided. 

3.  Sequence  is  important  with  unfamiliar  materials.  Students  who 
are  familiar  with  materials  will  learn  regardless  of  order  of 
presentation.  But  as  material  becomes  increasingly  unfamiliar  to 
the  student,  the  importance  of  sequence  increases. 

4.  Sequence  is  important  with  non-redundant  materials.  Some 
instructional  materials  are  especially  redundant,  stating  important 
points  over  and  over  again.  Sequencing  is  not  especially 
important  with  these  materials,  because  the  student  can  pick  up 
the  second  time  anything  he  has  missed  the  first  time. 

Following  are  guidelines  for  sequencing  objectives  with  dependent,  supportive, 
or  independent  relationships,  and  TLOs. 

Sequencing  Objectives  With  Dependent  Relationships 

In  order  to  sequence  learning  objectives  (LOs)  that  support  a  TLO,  review 
each  LO  to  determine  if  certain  skills  and  knowledges  are  prerequisites  for 
others.  Ask  the  question,  "What  other  skills  or  knowledges  must  the  student 
have  in  order  to  accomplish  this  task?"  The  answer  to  this  question  will  help 
to  identify  the  proper  sequence  for  prerequisite  objectives.  The  simpler 
prerequisite  objectives  should  be  sequenced  before  the  higher-level  (complex) 
objectives  of  which  they  are  a  part.  For  example,  before  a  person  can 
multiply,  he  must  be  able  to  add.  Learning  how  to  add  is  a  prerequisite  for 
multiplying,  and  must  be  taught  first. 

Sequencing  Objectives  With  Supportive  Relationships 

1)  Arrange  objectives  with  supportive  relationships  as  dose  together 
in  the  sequence  as  practical  so  that  optimum  transfer  of  learning 
can  take  place. 


2)  Arrange  objectives  dose  together  if  the  conditions  under  which 
the  learning  objectives  are  carried  out  are  identical  or  similar, 


and  if  the  conditions  are  difficult  or  expensive  to  produce  at 
random  times.  For  example,  if  several  learning  objectives  have 
conditions  such  as  "at  night,"  "on  muddy  terrain,"  or  "when  flying 
at  an  altitude  of  20,000  feet,"  you  will  probably  wish  to  place 
the  "at  night"  learning  objectives  together,  the  "on  muddy 
terrain"  learning  objectives  together,  etc. 

3.  Arrange  objectives  dose  together  if  a  particular  piece  of 

equipment  must  be  available  in  order  to  accomplish  a  group  of 
learning  objectives,  and  if  you  are  not  likely  to  have  continuous 
access  to  that  equipment  you  will  probably  wish  to  group  the 
learning  objectives  that  use  that  particular  equipment.  In 
addition,  you  will  want  to  keep  the  order  of  the  group  of 
learning  objectives  within  the  total  program  as  flexible  as 
possible.  For  example,  if  the  learning  objective  is  to  "perform 
maintenance  on  helicopter,"  you  will  have  to  wait  until  the 
helicopter  is  available  for  maintenance  before  you  can  accomplish 
the  objective. 

Sequencing  Objectives  With  Independent  Relationships 

Learning  objectives  with  independent  relationships  may  be  arranged  in  any 
sequence.  Any  of  the  guidelines  listed  for  sequencing  learning  with  a 
supportive  relationship  may  albe  be  used  for  sequencing  objectives  with 
independent  relationships.  However,  with  the  latter,  you  need  not  be 
concerned  about  locating  the  learning  objectives  dose  together  in  the 
sequence  unless  condition  or  equipment  constraints  indicate  otherwise. 

Sequencing  TLOs 

* 

The  same  guidance  for  sequencing  LOs  applies  to  the  sequencing  of  TLOs. 

For  example,  one  terminal  LO  might  be  to  operate  a  certain  piece  of 
equipment  while  another  terminal  LO  might  be  to  instruct  others  in  the 
proper  operation  of  the  equipment.  Most  likely  a  dependent  relationship 
probably  exists  between  these  two  terminal  learning  objectives;  therefore,  you 
would  place  "operate  equipment"  before  "instruct  others."  The  guidelines  for 
sequencing  supportive  and  independent  TLOs  are  likewise  the  same  as  given 
for  LOs. 


3.5.2  Course  Lessons  Sequenced  Improperly  (Question  76). 

The  sequence  of  lessons  within  the  course  did  not  follow  the  natural  order  of 
learning  inherent  in  those  lessons.  Lessons  that  teach  tasks  which  are 
subordinate  skills  to  tasks  taught  in  other  lessons  should  be  taught  first.  In 
general,  the  principles  for  sequencing  lessons  are  the  same  as  those  for 
sequencing  objectives  within  a  lesson.  Guidelines  for  accomplishing  the  latter 
are  contained  in  the  recommendations  for  the  previous  question. 
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3.6  PRESENTATION:  Clarity  and  Comprehensibility 


THE  PROBLEM.  Instruction  was  not  presented  clearly.  Presentation 
components  were  difficult  to  identify;  written  or  spoken  material  was  of  poor 
technical  quality;  the  wording  was  too  difficult  for  students  to  understand;  or 
the  instructors  speech  or  narration  was  difficult  to  listen  to. 

EVIDENCE  for  unclear  or  incomprehensible  instructional  presentations  will 
come  from  the  following  worksheets: 

Cl.l  Observation  of  Classroom  Instruction 

C1.2  Observation  of  Demonstrations 

C1.3  Observation  of  Isolated  Practice 

C1.7  Observation  of  Integrated  Practice 

Question  69.  Are  all  PRESENTATION  COMPONENTS  separated  and 
identified? 

Question  70.  Is  the  technical  quality  of  written  or  spoken  material 
adequate? 

Question  71.  Is  the  wording  of  written  or  spoken  material  easy  for  the 
students  to  understand? 

Question  72.  Is  the  instructor’s  presentation  or  the  narration  easy  to 
listen  to? 

% 

Question  91.  Is  the  speed  of  presentation  appropriate? 

RECOMMENDATIONS  for  training  program  modification. 

3.6.1  Presentation  Components  Not  Separated  or  Identified  (Question  69). 

Statements,  examples,  and  practice  components  were  not  separated  from  the 
rest  of  instruction,  or  identified  so  the  student  knows  what  they  are,  and  can 
locate  them.  Each  component  must  be  separated  from  the  rest  of  the 
instruction.  This  helps  the  student  find  the  main  idea.  When  the  component 
is  separated,  the  key  points  stand  out,  and  are  not  buried  in  the 
presentations.  There  are  several  ways  to  accomplish  this  goal. 

1.  Set  off  the  component  with  a  box. 

2.  Use  a  different  color  or  type  face,  or  underline. 

3.  Place  on  a  separate  page,  or  in  a  special  place  on  the  page. 

4.  For  audio,  movies,  or  lectures,  pause  before  introducing  the 
component. 

Each  component  must  also  be  identified.  After  the  component  is  separated, 
the  student  should  be  told  what  it  is.  This  permits  the  student's  attention  to 
be  focused  on  the  key  points  and  their  application,  rather  than  the  student 


trying  to  become  generally  familiar  with  everything  in  the  instruction.  Labels 
can  be  used  to  identify  different  components: 

Definition  of  ...  Example  Practice 

Procedure  for  ...  Demonstration  Test  Yourself 

Key  Point: 

3.6.2  Technical  Quality  of  Material  Inadequate  (Question  70). 

The  technical  quality  of  written  or  spoken  material  was  inadequate.  Whether 
text  appears  in  print,  in  some  other  visual  medium,  or  is  spoken  in  a  lecture 
or  demonstration,  it  should  meet  minimum  standards  of  technical  quality. 

1.  Each  component  should  be  directed  at  the  primary  audience,  e.g., 
trainees  rather  than  instructors  or  subject  matter  experts  with 
considerable  MOS  experience. 

2.  The  instruction  should  be  performance  oriented  rather  than  topic 
oriented.  All  presentations  should  be  performance  oriented  in 
two  ways.  First,  a  presentation  should  be  directly  related  to 
what  the  student  will  do  after  the  course,  i.e.,  REMEMBER  THE 
JOB.  All  too  often,  presentations  take  the  form  of  "everything 
you  wanted  to  know  about  ...",  with  no  information  about  what 
the  job  is  really  like,  or  why  the  information  is  important. 
Instead,  presentations  should  be  written  in  terms  of  the  job  to  be 
done.  Orienting  instruction  to  job  performance  is  done 
differently  for  different  presentation  components.  Statements 
give  essential  information,  and  are  not  job  oriented  by 
themselves.  The  job  orientation,  therefore,  is  given  by  providing 
additional  information  about  the  job  context  in  which  the 
statement  will  be  applied.  Examples  and  practice  items  are  job 
oriented  if  they  are  taken  from  actual  job  situations. 

Second,  a  presentation  should  also  orient  the  student  to  the 
performance  expected  during  the  instruction.  This  means  that 
the  student  must  be  told  what  is  in  the  instruction,  how  best  to 
study  it,  and  what  the  tests  will  be  like.  This  is  often 
accomplished  by  giving  the  objective(s)  at  the  beginning  of  the 
presentation.  Unfortunately,  objectives  are  not  usually  written  to 
communicate  directly  to  the  student.  They  can  be  confusing, 
because  they  contain  unfamiliar  words  and  topics,  and 
complicatea  sentences.  Instead,  the  instructional  orientation 
should  tell  the  student  what  is  in  the  presentation,  what  he 
should  do  with  the  information  (e.g.,  memorize  it),  and  how  he 
will  be  tested. 

3.  Main  points  should  stand  out.  They  should  not  get  lost  in  detail, 
so  that  secondary  points  appear  equal  to  main  points. 

Components  should  not  be  too  wordy,  either  because  there  is  too 
much  detail  (more  than  the  essentials  needed  to  perform  the 
job),  or  because  the  material  is  redundant  (with  information  being 
needlessly  repeated). 
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4. 


Information  should  be  presented  in  an  orderly  manner  so  that  it 
is  not  confusing.  There  should  not  be  redundancy  between 
paragraphs  or  ideas,  missing  information,  scattered  information 
(information  about  the  same  main  point  not  in  one  place),  or 
remote  references  (references  to  text  or  illustrations  in  other 
places). 

5.  Text  should  not  have  a  run-together  format.  A  run-together 

format  is  a  solid  mass  of  print.  The  reader  cannot  easily  scan 
it.  He  cannot  easily  identify  and  lift  out  the  separate  points. 

The  solution  to  a  run-together  format  is  to  break  the  text  up 
into  visual  "chunks"  which  group  the  information  logically. 

Proper  visual  chunking  lets  the  reader  scan  and  identify  the 
separate  points  easily. 

Problems  with  text  materials  should  be  referred  to  the  training 
of  technical  manual  developers.  A  complete  Instructor  Training 
Course  should  include  techniques  for  conducting  clear  and 
comprehensible  instructional  presentations. 

3.6.3  Too  Many  Difficult  Words  or  Long  Sentences  (Question  71). 

The  wording  of  written  or  spoken  material  was  difficult  for  the  student  to 
understand.  The  purpose  of  instruction  is  to  help  the  student  to  reach  the 
terminal  learning  objective  and  its  prerequisite  learning  objectives.  The  more 
effectively  the  instructor  or  text  can  communicate  with  him,  the  more  likely 
he  is  to  learn  from  the  instruction.  Therefore  it  is  important  NOT  to  exceed 
the  student's  reading  level  (or  vocabulary  level  if  instruction  is  lecture  or 

audio-visual  only). 

% 

Writing  that  is  easy  to  read  is  usually  easy  to  understand.  It 
should  use  words  with  which  the  student  is  familiar,  and  put 
them  into  fairly  short,  uncomplicated  sentences. 

The  following  list  of  guidelines  for  lowering  the  reading  difficulty 
level  of  instruction  is  divided  into  two  sections:  (1)  Word 
Selection  and  (2)  Sentence  Structure. 

WORD  SELECTION 

1.  Use  SHORT,  FAMILIAR  words  when  possible. 

POOR  CHOICE  BETTER  CHOICE 

accordingly  so 

assistance  help,  aid 

facilitate  help,  ease 

utilization  use 
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feasible 


possible 


implement  carry  out 

Reading  studies  show  that  short,  familiar  words  tend  to  communicate 
better. 

2.  Use  CONCRETE,  SPECIFIC  words. 

Poor  Choice  Better  Choice 

aircraft  B-52 

Majority  of  soldiers  95%  of  soldiers 

physical  needs  hunger  and  thirst 

weapon  M-16 

Abstract  words  mean  different  things  to  different  people.  Therefore, 
they  can  make  the  intended  meaning  unclear.  Concrete  words 
communicate  more  easily  and  precisely. 

3.  Use  NON-TECHNICAL  words. 

Poor  Choice  Better  Choice 

davical  collar  bone 

NCOIC  non-commissioned  officer  in  charge 

terrain  ground 

experimentation  test 

When  a  technical  word  cannot  be  replaced  by  a  non-technical 
word  and  the  word  is  unfamiliar  to  the  student,  the  technical 
word  should  be  explained  in  a  separate  sentence  either  directly 
before  or  directly  after  its  first  appearance. 

Of  course,  if  the  students  have  a  technical  background  in  the 
area,  technical  terms  are  appropriate. 

In  the  case  of  military  abbreviations,  it  is  sufficient  to  write  out 
the  complete  term  followed  by  its  abbreviation  in  parentheses 
the  first  time;  and  to  use  the  abbreviation  every  time 
afterwards. 

EXAMPLE:  The  non-commissioned  officer  in  charge  (NCOIC) 
gave  the  order.  The  NCOIC  told  the  men  to  shine  their  boots. 


4. 


Use  ACTIVE  voice  verbs. 


Poor  Choice; 

The  statement  must  be 
updated  every  three  months. 

Poor  Choice: 

The  truck  will  be  driven 
by  Specialist  Jones. 


Better  Choice: 

You  must  update  the  statement 
every  three  months. 

Better  Choice: 

Specialist  Jones  will  drive 
the  truck. 


This  guideline  does  not  mean  that  one  must  never  use  the 
passive  voice.  However,  it  should  be  avoided  in  instructions  and 
procedures,  as  in  the  first  example;  otherwise,  the  job  may  not 
get  done!  ("Who  is  to  update  the  statement,"  is  important  to 
your  meaning.) 


5.  Avoid  changing  action  verbs 

Poor  Choice: 

In  the  Army,  careful  initial 
selection  and  classification  are 
important  procedures  in 
eliminating  maladjusted  personnel. 

When  verbs  are  changed  into  nouns 

to  whom." 
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SENTENCE  STRUCTURE 

1.  Use  short,  simple  sentences. 

Poor  Choice;  Be 


into  nouns. 

Better  Choice: 

If  the  Army  selects  and 
classifies  personnel  carefully, 
it  will  have  fewer  maladjusted 
people. 

it  is  difficult  to  tell  "who  did  what 


Choice; 


In  the  case  of  the  habitual 
offender,  there  is  nothing  to 
do  but  remove  him  from  the 
Service;  needless  to  say,  he  is 
what  might  be  termed  ineffec¬ 
tual  as  a  soldier. 


Remove  the  habitual  offender 
from  the  Service.  He  is  not 
an  effective  soldier. 


2.  Avoid  strings  of  prepositional  phrases.  Example;  The  hand  of 
the  man  on  the  roof  of  the  house. 

3.  Shorten  prepositional  phrases. 


Poor  Choice: 


Better  Choice: 


in  a  manner  similar  to 

like;  as 

with  reference  to 

about 

in  view  of  the  fact 

since;  because 

with  due  regard  for 

for 

in  a  situation  in  which 

when 

4.  Avoid  wasteful  words. 

Poor  Choice: 

Better  Choice: 

are  desirous  of 

want  to 

gained  from  the  following 

from 

source 

It  is  recommended  that  con¬ 
sideration  be  given  to  ... 

Consider  ... 

is  responsible  for  selecting 

selects 

make  provision  for 

provide 

take  appropriate  measures 

act;  do 

the  fullest  possible  extent 

the  most 

afford  an  opportunity 

allow 

3.6.4  Instructor's  Presentation  or  Narration  Difficult  to  Listen  To  (Question 
72E 

The  verbal  portion  of  the  instruction  was  not  easy  to  listen  to.  * 

For  spoken  material  to  be  easy  to  listen  to,  its  pace  and  pitch  should  be 
varied.  It  should  not  continue  at  the  same  rate  or  at  the  same  pitch  until  it 
becomes  monotonous.  Also,  repetition  of  short  patterns  of  pace  and/or  pitch 
which  give  speech  a  "sing-song"  effect  should  be  avoided.  In  addition,  speech 
should  be  understandable.  It  should  not  be  too  soft;  speakers  should  not  have 
speech  impediments  or  heavy  accents,  nor  should  they  mumble.  Instruction  on 
verbal  presentation  techniques  are  part  of  the  Instructor  Training  Course  that 
all  instructors  should  have  attended.  If  instructors  have  not  attended  an 
Instructor  Training  Course,  they  should  be  scheduled  at  the  earliest  possible 
time. 

3.6.5  Instructor  Presented  Material  Too  Fast  or  Too  Slowly  (Question  91). 

The  instructor  presented  statements,  examples,  or  help  so  fast  that  students 
had  trouble  following  the  presentation,  or  so  slowly  that  students  became 
bored.  While  a  rapid  presentation  rate  is  all  right  for  review  when  trainees 
have  already  mastered  the  material,  trainees  new  to  the  task  need  to  have 
the  material  presented  slowly  so  they  can  follow  along  without  getting  lost. 
Clues  to  problems  with  presentation  speed  will  come  from  observations  made 
by  the  training  observer.  If  instructors  backtrack  to  repeat  information  just 
presented,  they  are  already  aware  of  the  problem. 


If,  on  the  other  hand,  the  instructor  slows  his  presentation  down  to  a  "crawl" 
for  the  benefit  of  the  poorest  performer,  he  runs  the  risk  of  boring  the  rest 
of  the  trainees.  This  trainee  boredom  may  carry  over  to  the  next  training 
session.  It  hurts  trainee  motivation,  a  fragile  commodity  at  best.  Trainees 
who  cannot  keep  up  with  the  rest  of  the  class  should  be  scheduled  for 
remediation  outside  of  class  time. 
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3.7  PRESENTATION:  Examples 


THE  PROBLEM.  Effective  examples  were  not  included  in  the  instructional 
presentation.  Examples  or  non-examples  were  inadequate  or  there  were  an 
insufficient  number  of  examples,  or  example  help  was  inadequate.  For 
concept  objectives,  examples  were  not  sequenced  from  easy  to  hard,  or  non¬ 
examples  were  not  included.  (Demonstrations,  which  are  examples  of 
procedures,  are  covered  in  the  section  on  DEMONSTRATIONS). 

EVIDENCE  for  ineffective  examples  will  come  from  Worksheets  Cl. I 
Observation  of  Classroom  Instruction,  and  C1.2  Observation  of 
Demonstrations. 

Question  37.  Are  EXAMPLES  and  NON-EXAMPLES  adequate? 

Question  38.  Is  EXAMPLE  help  adequate? 

Question  39.  Are  EXAMPLES  sequenced  from  easy  to  hand? 

Question  40.  Are  there  enough  EXAMPLES? 

Question  41.  Are  NON-EXAMPLES  included? 

RECOMMENDATIONS  for  training  program  modification. 

3.7.1 

Students  were  not  clearly  told  or  shown  how  a  statement  of  a  concept, 
procedure,  rule,  or  principle  applies  in  specific  cases.  For  concept  objectives, 
examples  should'show  all  critical  characteristics  required  for  classification; 
non-examples  should  show  the  absence  of  critical  characteristics.  For 
procedure  objectives,  application  of  the  procedure  must  be  shown  and  steps 
must  be  shown  in  the  correct  order.  For  rule  objectives,  application  of  each 
step  or  branching  decision  must  be  shown  in  the  correct  order.  For  principle 
objectives,  interpretation  or  prediction  based  on  causes,  effects,  and 
relationships  must  be  shown.  Additional  guidance  on  the  design  of  adequate 
examples  and  non-examples  follows. 

3.7.2 

Example  help  did  not  enhance  student  understanding.  Help  is  applied  in 
different  ways  for  different  content  types.  Some  types  of  help  recommended 
for  each  content  type  are  given  below: 

CONCEPTS: 

Highlight  the  critical  characteristics  of  an  example. 

Explain  why  or  why  not  something  is  classified  as  a  member  of  a 
concept.  Show  the  use  of  a  checklist  or  heuristic  to  help  classify. 

Simplify  early  examples,  e.g.,  use  line-drawings  instead  of  complicated 
photographs. 


PROCEDURES  OR  RULES: 


Explain  why  each  step  is  done. 

Explain  why  each  step  is  important. 

Give  additional  information  about  how  to  perform  the  task. 

Give  additional  information  about  how  to  know  if  it  was  performed 
incorrectly. 

Give  flowcharts,  tables,  etc. 

PRINCIPLES: 

Highlight  important  features. 


Simplify  the  relevant  information  from  the  case  study  in  which  it  is 
embedded. 

Use  logical  representations  of  the  IF-THEN  relationships. 

Give  additional  information  about  why  the  principle  applies,  or  why  it 
doesn't. 

Give  hints  as  to  how  to  analyze  problems. 

3.7.3  Examples  Sequenced  Improperly  (Question  39). 

s 

Examples  for  Concept  objectives  were  not  sequenced  from  easy  to  hard. 

Early  examples  should  be  simplified.  In  this  way  the  learner  will  be 
positively  motivated  by  success  in  understanding  the  easier  examples,  and  at 
the  same  time  be  challenged  by  the  more  difficult  ones.  Estimates  of 
example  difficulty  should  be  in  reference  to  the  students,  not  the  instructor. 

A  good  way  to  determine  example  difficulty  is  to  provide  a  sample  of 
students  from  the  target  population  with  the  definition  of  the  concept  and  a 
list  of  randomly  sequenced  examples  and  non-examples.  The  students  are  then 
asked  to  classify  each  one.  Difficulty  ratings  are  then  calculated  for  each 
example  or  non-example  by  dividing  the  number  of  correct  responses  by  the 
number  of  students  in  the  sample.  The  results  of  the  analysis  can  then  be 
ordered  in  terms  of  frequency  of  correct  responses  along  a  continuum. 
Examples  which  more  than  70%  of  the  students  correctly  identify  are  easy, 
while  examples  which  less  than  30%  of  the  students  correctly  classify  are 
considered  difficult.  Examples  which  fall  in  between  (30-70%  correctly  identi¬ 
fied)  are  of  medium  difficulty.  Examples  and  non-examples  can  then  be  taken 
from  this  example  pool  and  presented  in  order  of  difficulty  (easy  to  hard). 

3.7.*  Insufficient  Number  of  Examples  (Question  40). 

There  were  not  enough  examples  to  cover  the  content  area  thoroughly.  For 
Concepts  there  should  be  at  least  one  example/non-example  pair  for  each 
critical  characteristic.  For  Procedures  or  Rules,  there  should  be  one  example 
for  each  important  step,  and  examples  for  all  possible  branches  or  decisions. 
More  detailed  guidance  for  Concept  and  Rule  examples  is  provided  below: 
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3.7.5  Examples  for  Concepts 


In  teaching  a  concept,  the  instructor  (or  instructional  material)  should: 

1.  Show  a  wide  enough  range  of  examples  to  prevent 

under  generalization  (failure  to  recognize  an  example  as  an 
example).  To  do  this  the  instructor  should  present  a  number  of 
examples  that  all  have  the  critical  characteristics  but  which  vary 
widely  on  irrelevant  characteristics.  This  shows  the  student  that 
legitimate  examples  don't  always  look  the  same.  It  minimizes 
the  student's  tendency  to  not  recognize  legitimate  examples 
because  they  look  a  little  different.  The  instructor  should  also 
be  aware  that  certain  concepts  have  critical  characteristics  that 
have  a  permissable  range  of  values  (e.g.,  one  critical 
characteristic  of  an  airplane  is  that  it  has  one  or  more  engines). 
It  is  important  in  such  cases  that  the  example  set  samples  this 
range  of  acceptable  values. 

2.  Show  each  example  paired  with  a  corresponding,  matched  non¬ 
example  to  prevent  overgeneralization  (tendency  to  identify 
instances  as  examples  which  are  not).  The  nonexample  should  be 
identical  with  the  example  on  irrelevant  characteristics. 

However,  it  will  be  lacking  one  or  more  of  the  critical 
characteristics.  Pairing  focuses  the  student's  attention  on  the 
critical  characteristics.  This  helps  minimize  any  tendency  to 
identify  a  nonexample  as  a  legitimate  example  by  showing  the 
likely  nonexamples  and  pointing  out  why  they  are  not  examples. 


This  table  summarizes  what  the  instructor  should  strive  for  in  his 
series  of  example-nonexample  pairs: 

Examples  for  Rules 

The  purpose  of  presenting  a  series  of  examples  for  a  rule  is  to  show  the 
application  of  the  rule  to  a  wide  range  of  cases.  By  carefully  choosing  the 
examples  to  be  presented  in  divergent  example  sequences,  a  very  wide  range 
of  applications  can  be  shown  with  very  few  examples.  Examples  are 
divergent  or  significantly  different  from  each  other  to  the  degree  that  each 
example  shows  the  rule  being  applied  to  a  different  set  of  data.  The  "data" 
are  the  pieces  of  information  manipulated  by  the  rule.  They  are  the  inputs 
which  are  used  during  the  rule's  operation.  These  data  or  input  vary  in  form 
(or  VALUE)  from  example  to  example.  The  instructor  should  select  values  for 
each  type  of  data  that  students  are  likely  to  encounter,  and  should  adequately 
sample  the  possible  range  of  values.  If  the  possible  values  form  a  continuum, 
the  instructor  can  pick  values  at  the  beginning,  middle,  and  end  of  the 
continuum  to  use  in  the  items.  Note  that  only  the  instructor  or  the 
developer  of  instruction  can  determine  whether  all  the  necessary  data  types 
and  values  have  been  sampled.  This  determination  is  based  on  subject  matter 
expertise  and  thus  cannot  be  made  by  someone  who  does  not  know  the 
content. 
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For  Concept  objectives,  non-examples  were  not  included  in  the  instruction. 
Examples  should  be  matched  with  non-examples  according  to  the  presence  or 
absence  of  each  critical  characteristic.  When  teaching  concepts,  it  is 
important  to  present  a  range  of  examples  to  insure  complete  learning  of  the 
concept.  By  showing  paired  examples  and  nonexamples,  the  instructor  can 
help  the  student  focus  attention  on  the  critical  characteristics  shown  in  the 
examples  and  missing  in  the  nonexamples. 

One  condition  must  be  met  for  an  example  and  nonexample  of  a  concept  to 
be  matched:  the  useful  irrelevant  characteristic  values  of  the  concept 
present  in  the  example  should  be  identical  to  those  in  the  nonexample.  In 
this  way  the  critical  characteristics  not  present  in  the  nonexample  are 
isolated  in  the  example,  and  are  the  only  things  different.  Considering  an 
example  should  make  this  clear. 

These  are  the  critical  characteristics  of  the  concept  SQUARE: 

A  square  has 

1.  Four  equal-length  sides. 

2.  Four  right  internal  angles. 

Here  is  a  MATCHED  example-nonexample  pair: 

This  is  a  square.  This  is  not  a  square. 

□  a 

There  are  many  irrelevant  characteristics  of  geometric  figures  to  draw  from. 
A  few  of  them  which  might  be  useful  to  show  a  variety  of  examples  which 
the  student  might  encounter  include: 

1.  Shading/coloring 

2.  Size 

3.  Line  thickness 

These  two  figures  are  identical  in  as  many  of  these  irrelevant  characteristics 
as  possible.  Their  respective  sizes  are  about  the  same.  Their  lines  are  of 
equal  thickness  and  they  are  not  shaded.  Therefore,  the  main  difference 
between  them  is  the  absence  of  one  critical  characteristic  having  four  right 
internal  angles  (the  square  has  four  right  internal  angles,  the  nonsquare  does 
not). 

Here  are  some  example-nonexample  pairs  which  are  UNMATCHED: 


This  is  a  square.  This  is  not  a  square. 


These  figures  differ  in  the  irrelevant  characteristic  "shading."  Therefore,  they 
are  not  as  well  matched  as  they  could  be.  They  would  be  better  matched  if 
the  square  was  also  shaded. 

This  is  a  square.  This  is  not  a  square. 
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These  figures  differ  in  the  irrelevant  characteristic,  "size."  They  would  be 
better  matched  if  they  were  more  nearly  the  same  size. 

This  is  a  square.  This  is  not  a  square. 

□  o 

These  figures  differ  in  the  irrelevant  characteristic  "line  thickness."  A  naive 
learner  might  conclude  that  the  important  difference  between  a  square  and 
nonsquare  has  something  to  do  with  line  thickness.  They  should  be  matched 
for  line  thickness,  since  it  is  a  irrelevant  characteristic. 

Note:  When  a  irrelevant  characteristic  is  NOT  identical  in  both  the 
example  and  nonexample,  the  learner  might  mistakenly  assume 
that  the  irrelevant  characteristic  is  a  critical  characteristic. 

The  purpose  of  matched  example-nonexample  pairs  is  to  isolate 
critical  characteristics. 

Here  is  a  properly  MATCHED  example-nonexample  for  the  concept  SQUARE: 


This  is  a  square.  This  is  not  a  square. 

□  o 


These  figures  are  identical  in  irrelevant  characteristics,  number  of  sides,  tine 
thickness,  and  lack  of  shading.  Since  as  many  irrelevent  characteristics  as 
possible  are  identical,  the  instance  satisfies  the  criteria  of  a  matched  example- 
nonexample  pair. 
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3.8  PRESENTATION:  Media 


THE  PROBLEM.  Media  used  in  training  were  inappropriate  for  the 
task/content  level  of  the  objectives  in  the  lessons  or  did  not  provide  all  of 
the  necessary  stimuli  required  by  the  objectives. 

EVIDENCE  for  inappropriate  or  inadequate  media  will  come  from  Worksheet 
B3  Product  Evaluation  -  Evaluate  Presentation. 

Question  77.  Are  the  media  appropriate  for  the  objectives? 

Question  78.  Can  the  media  used  provide  all  necessary  stimuli  reauired 
by  the  OBJECTIVES'7 

RECOMMENDATIONS  for  training  program  modification. 

3.8.1  Inappropriate  Media  (Question  77). 

Media  used  for  a  group  of  objectives  were  inappropriate  for  the  task/ con  tent 
level  of  those  objectives.  The  specific  problem  with  the  media  used  will  have 
been  indicated  in  the  "Description  Column"  of  Worksheet  B3.  These  mismatch 
problems  are  listed  below  along  with  recommendations  for  their  solution: 

No  real  practice.  The  task  level  is  Use  and  the  consequences  of  error  are 
serious  (when  the  graduate  is  first  required  to  perform  the  task  on  the  job). 
The  media  used  cannot  provide  the  realism  needed  to  practice  the  task 
adequately  under  these  conditions.  Recommend  that  one  of  the  following 
media  be  used  instead:  (media  definitions  may  be  found  in  Appendix  B). 

•  Large  equipment 

•  Portable  equipment 

•  Simulator 

Incomplete  information.  The  task  level  is  Remember  and  the  instructional 
setting  is  self-instruction.  The  media  used  would  not  present  enough 
information  without  the  aid  of  an  instructor.  Recommend  that  one  of  the 
following  media  be  used  instead: 

•  Audio 

•  Filmstrip 

•  Motion  picture 

•  Printed  text 

•  Slide/tape 

•  Training  aid 

•  TV  cassette 

•  Computer 

•  Programmed  text 

•  Interactive  TV 

•  Training  device  (supplemented  with  printed  text) 

•  Portable  equipment  (supplemented  with  printed  text) 
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No  feedback.  The  task  level  is  Use  Mental  Skills,  i.e.,  concepts,  procedures, 
rules  or  principles  which  are  performed  primarily  in  the  student's  mind.  (A 
skill  may  have  both  mental  and  motor  skills  associated  with  it,  but  if  the 
motor  skills  are  so  simple  that  anyone  who  knew  the  mental  skills  could 
perform  them  without  training,  the  overall  skill  should  be  considered  a  mental 
skill).  The  instructional  setting  is  self-instruction.  The  media  used  cannot 
provide  corrective  feedback  without  an  instructor.  Recommend  that  one  of 
the  following  media  be  used  instead: 

•  Computer 

•  Programmed  text 

•  Training  device 

•  Motion  picture 

•  Slide/tape 

•  TV  cassette 

•  Filmstrip 

•  Audio 

No  hands-on  practice.  The  task  level  is  Use  Motor  Skills,  i.e.  procedures 
which  require  the  smooth  timing  of  muscular  movements,  (all  motor  skills 
have  a  mental  component,  but  with  motor  skills  the  physical  activity  must  be 
practiced  in  order  to  achieve  proficiency).  The  media  used  cannot  provide 
the  necessary  motor  manipulations. 

If  the  instructional  setting  is  self-instruction,  recommend  that  one  of  the 
following  media 'be  used  instead: 

•  Training  device  (supplemented  with  printed  text) 

•  Portable  equipment  (supplemented  with  printed  text) 

If  the  setting  is  instructor-led  instruction,  one  of  the  following  media  should 
be  used  along  with  the  instructor: 

•  Portable  equipment 

•  Large  equipment 

•  Training  device 

•  Simulator 

Note  that  all  of  the  media  recommended  for  each  situation  are  not 
necessarily  the  best  media  in  terms  of  costs  and  practicality;  however,  they 
can  provide  the  necessary  learning  characteristics  for  these  conditions. 
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3.8.2  Media  Lacks  Required  Stimuli  (Question  78). 

The  media  used  did  not  display  the  specific  stimuli  required  for  students  to 
learn  the  objective.  The  specific  stimulus  requirement  will  probably  be  one 
of  the  twenty  media  stimulus  capabilities  listed  below. 

Visual  Form 


1.  Visual  Alphanumeric  -  words,  numbers  and  other  symbols 
presented  graphically. 

2.  Visual  Pictorial,  Plane  -  a  two-dimensional  image,  such  as 
representation  in  the  form  of  a  photograph  or  drawing. 

3.  Visual  Line  Construction  Plane  -  a  two-dimensional  figure  made 
of  lines,  such  as  a  mathematical  curve  or  graph. 

4.  Visual  Object,  Solid  -  a  three-dimensional  image  or  reality  that 
is  viewed  from  exterior  perspectives. 

5.  Visual  Environment  -  A  three-dimensional  image  or  reality  that  is 
viewed  from  inside. 

Visual  Movement 


6.  Visual  Still  -  a  static  visual  field,  such  as  a  photograph,  drawing, 
or  printed  page. 

7.  Visual  Limited  Movement  -  a  basically  static  visual  field  with 
elements  that  can  be  made  to  move,  such  as  an  animated 
transparency  or  simple  panel  with  switches  that  move. 

8.  Visual  Full  Movement  -  a  visual  field  in  which  all  elements  can 
move,  such  as  a  motion  picture,  flight  simulator,  or  operational 
aircraft. 

9.  Visual  Cyclic  Movement  -  a  visual  field  which  moves  through  a 
fixed  sequence  and  then  repeats  the  sequence  in  a  repetitive 
manner,  such  as  a  film  loop. 

Visual  Spectrum 

10.  Black  and  White  -  a  visual  field  composed  of  black  and  whilte 
dements,  such  as  the  printed  page  or  line  drawings. 

11.  Gray  Scale  -  a  visual  field  composed  of  black,  white  and 
continuous  gradations  of  gray,  such  as  a  black  and  white 
photograph  or  television  picture. 


12. 


Color  -  a  visual  field  composed  of  various  segments  of  the  visual 
spectrum,  such  as  color  television  or  motion  pictures. 


Scale 


13.  Exact  Scale  -  actual  visual  field  or  a  one-to-one  replication  of  a 
field  such  as  a  full-sized  mock-up,  simulator,  or  operational 
system. 

14.  Proportional  Scale  -  a  representation  of  reality  in  other  than  full 
scale,  such  as  a  scaled  model  map  or  photograph. 

Audio 

15.  Voice  Sound  Range  -  a  limited  quality  of  sound  which  enables 
spoken  words  to  be  used  as  the  medium  of  communication,  but 
not  suited  to  more  demanding  tasks,  such  as  music  or  sound 
recognition  exercises. 

16.  Full  Sound  Range  -  a  quality  of  sound  reproduction  that  contains 
all  the  significant  elements  of  the  sound  and  is  suited  to  the 
demanding  task  of  sound  recognition  exercises. 

17.  Ambient  Sounds  -  a  complex  sound  environment  with  sound  from 
various  sources  and  from  various  directions,  including  background 
noise  and  task  significant  sounds. 

Other 

18.  Tactile  Cues  -  signals  received  through  the  sense  of  touch, 
including  sensations  related  to  texture,  size  or  shape. 

19.  Internal  Stimulus  Motion  Cues  -  the  sensations  felt  by  a  person 
when  he  moves  his  arm,  leg,  fingers,  etc. 

20.  External  Stimulus  Motion  Cues  -  the  sensations  felt  by  a  person 
when  moved  by  some  outside  force  in  such  a  way  that  his  body 
experiences  roll,  pitch,  yaw,  heave,  sway  and/or  surge. 

If  the  media  employed  does  not  have  the  required  stimulus  capability, 
recommend  that  the  objective  be  taught  in  another  medium  which  can 
accommodate  the  requirement.  For  example,  suppose  the  objective  involves 
giving  oral  orders  in  a  combat  situation.  This  objective  would  have 
requirements  for  voice  sound  range  and  ambient  sounds,  among  others.  Use 
of  a  job  performance  aid  or  typical  classroom  instruction  would  not  meet 
these  requirements.  In  this  case,  you  would  recommend  that  the  objective  be 
taught  instead  in  a  simulator  or  using  operational  equipment  in  a  realistic 
environment. 
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3.9  PRESENTATION:  Visuals 

THE  PROBLEM.  Visuals  were  not  easily  understood  by  the  trainees,  or  were 
not  used  effectively  or  at  all  during  the  instruction.  Visuals  should  enhance 
instruction.  If  they  do  not,  either  the  visuals  themselves  or  the  instructions 
for  their  use  should  be  changed.  Note  that  the  focus  here  is  on  visuals  used 
as  training  aids,  not  job  performance  aids.  Job  performance  aids  are  covered 
under  DEMONSTRATION/PRACTICE,  Questions  36,  51,  and  52). 

EVIDENCE  for  the  ineffective  use  of  visuals  will  come  from  Worksheets 
Cl.l  Observation  of  Classroom  Instruction,  and  Cl. 2  Observation  of 
Demonstrations. 

Question  73.  Is  the  instructor's  presentation  or  the  narration  supported 
by  visuals? 

Question  74.  Are  visuals  easily  understood? 

RECOM MEND ATIONS  for  training  program  modification. 

3.9.1  Verbal  Presentations  Not  Supported  oy  Visuals  (Question  73). 

In  order  to  maintain  attention  and  interest  and  to  aid  students  in  under¬ 
standing  the  material,  verbal  presentations  should  be  accompanied  by  visuals. 
Visuals  can  be  used  as  a  HELP  for  a  statement,  an  example,  or  feedback. 
Guidelines  for  visual  HELP  for  various  types  of  content  are  given  below. 

3.9.2  Visuals  as  HELP  for  Fact  Objectives 

Visuals  can  be  used  to  explain  information  which  was  initially  presented  in 
written  form.  For  example,  if  learners  are  asked  to  memorize  the  locations 
of  all  the  fire  extinguishers  in  a  building,  a  drawing  pinpointing  all  their 
locations  could  be  very  helpful. 

Visuals  as  HELP  for  Memory-Level  Objectives  (i.e..  Remember  Concept, 
Procedure  or  Rule! 


Associating  something  to  be  remembered  with  a  vivid,  visual  image  is  often 
effective.  In  such  cases  the  visual  should  be  as  unambiguous  as  possible. 

This  means  that  the  visuals  used  should  be  concrete  rather  than  symbolic  in 
nature  and  that  each  should  illustrate  some  particularly  distinctive  feature  of 
the  step  or  characteristic  to  be  memorized.  This  technique  is  particularly 
useful  if  the  content  (the  rule,  concept,  or  procedure  being  taught)  is 
inherently  visual  in  some  respects.  For  instance,  it  would  be  easy  to 
illustrate  the  characteristics  of  a  bicycle  (two  wheels,  pedal  powered)  because 
you  can  see  wheels  and  pedals.  These  characteristics  are  concrete.  However, 
it  would  be  harder  to  graphically  depict  the  characteristics  (i.e.,  lyric  nature, 
interrelationship  of  parts)  of  a  sonnet.  These  characteristics  are  abstract  and 
any  visual  developed  would  be  symbolic  in  nature.  Symbolic  visuals  are  often 
misinterpreted  and,  therefore,  are  less  useful  as  memory  aids. 

With  concepts,  this  visual  help  is  very  similar  to  the  technique  used  with 
USE/CONCEPT  objectives  of  showing  an  example  with  characteristics 
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IS. 


isolated.  However,  when  selecting  an  example  for  a  REMEMBER-CONCEPT 
objective,  select  an  example  which  is  most  likely  to  be  familiar  to  the 
audience,  one  with  the  least  number  or  least  confusing  values  for  irrelevant 
characteristics  and  with  easily  identified  critical  characteristics.  For 
instance,  if  teaching  a  REMEMBER-CONCEPT  lesson  about  bicycles,  you 
would  seLect  a  standard  street  bike  with  no  streamers,  no  hand  brakes,  a 
regular  seat.  When  designing  a  REMEMBER-CONCEPT  SEGMENT,  you  want 
to  avoid  confusing  the  learner  with  irrelevant  characteristics.  When  designing 
USE-CONCEPT  instruction,  you  may  deliberately  introduce  irrelevant 
characteristics  which  are  confusing,  to  help  learners  avoid  common  errors. 

Visuals  as  HELP  for  Use-Procedure  Objectives 

Visuals  or  illustrations  may  be  used  to  highlight  aspects  of  procedure  steps 
which  need  to  be  shown  visually.  Graphic  techniques  such  as  the  use  of 
arrows,  exploded  sections,  or  inserts  or  illustrations  may  clarify  confusing  or 
difficult  steps. 

For  example,  a  step  in  the  procedure  of  preparing  a  35-mm  camera  to  take 
pictures  might  be  to  "set  the  ASA/DIN  to  match  film  speed."  In  addition  to 
defining  "ASA/DIN"  and  film  speed,  you  might  also  want  to  show  a  diagram 
of  a  typical  camera  with  the  mechanism  for  setting  the  ASA /DIN  clearly 
shown. 

Visuals  as  HELP  for  Use-Rule  Objectives 

Graphics  techniques  such  as  arrows  may  be  used  for  pointing  out  particular 
aspects  of  rule  applications  to  whole  or  partial  examples. 

For  instance,  one  step  in  the  rule  for  calculating  the  molecular  weight  of  a 
compound  requiKes  counting  the  number  of  atoms  of  each  element  in  the 
compound.  The  help  for  that  step  might  show  the  following  examples,  using 
arrows  for  illustration. 

Visuals  Hard  to  Understand  (Question  74). 

Trainee  questions  and  comments  indicate  that  they  found  the  visuals  hard  to 
understand.  Such  difficulty  can  arise  when  the  visual  conflicts  with  something 
the  trainee  is  used  to  or  has  seen  before,  when  the  visual  does  not  fit  in  well 
with  the  instructional  sequences  or  when  it  is  poorly  designed.  The  overriding 
rule  here  is  that  a  visual  has  only  one  real  purpose,  to  aid  student  learning. 

If  it  does  not,  the  visual  and/or  instruction  need  to  be  redesigned.  It  is  not 
the  purpose  of  this  paragraph  to  detail  how  to  design  effective  visuals. 
Problems  alluded  to  here  should  be  described  to  training  development  and/or 
graphics  personnel. 

3.10  Presentation:  Open-ended  Probe  Item 

THE  PROBLEM.  Any  comment  on  training  may  be  aired  here,  involving 
training  environment,  presentation,  or  practice/demonstration.  The  TEE 
evaluator  will  have  to  determine  where  such  comments  fit,  and  how  best  to 
use  them.  In  this  type  of  question,  respondents  tend  to  be  bound  by  memory; 
the  first,  last,  and  most  unusual  situations  tend  to  stand  out.  One  way  to 
use  this  item  would  be  to  content  analyze  the  responses  by  grouping  them 
into  topics  areas.  If  one  topic  has  many  responses,  the  evaluator  could  check 
to  see  if  the  data  support  any  statements  made  by  the  students.  The  rating 
on  this  item  is  not  as  valuable  as  the  information.  Remember  that  the 
responses  are  subjective. 
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PRACTICE/DEMONSTRATION 


4.1  PRACTICE/DEMONSTRATION:  Practice  (General) 

THE  PROBLEM.  Practice,  defined  as  hands-on  performance  of  the  task, 
supervised  by  an  instructor  or  assistant  instructor,  during  training,  either  did 
not  happen  or  did  not  happen  as  it  should  have.  Practice  events  are  essential 
to  skill  development.  No  matter  how  good  lectures,  demonstrations,  "talk- 
throughs,"  or  "walk-throughs"  are,  trainees  will  not  learn  to  perform  tasks 
unless  they  practice  them.  It  is  essential  that  during  the  practice  event, 
every  trainee  practice  each  and  every  subtask  and  task,  until  performance  to 
the  training  standard  under  the  training  conditions  is  reached.  Performance 
to  standard  must  be  demonstrated  by  the  soldier  at  least  once  without 
assistance. 

EVIDENCE  for  missing  or  poor  practice  events  will  come  from  Worksheets 
Cl. 3  Observation  of  Isolated  Practice,  and  Cl. 7  Observation  of  Integrated 
Practice. 

Question  58.  Is  FINAL  PRACTICE  free  of  external  cues  or  help? 

Question  59.  Are  there  PRACTICE  items  for  each  TLO  or  all  of  its 
critical  parts/LOs? 

Question  60.  Is  there  a  PRACTICE  item  for  each  critical  part  of  each 
LO? 

Question  61.  Do  all  students  PRACTICE? 

Question  62.  Do  all  students  meet  the  required  STANDARDS  in  FINAL 
PRACTICE?. 

RECOMMENDATIONS  for  training  program  modification. 

4.1.1  Trainees  Helped  During  Final  Practice  (Question  58). 

It  is  entirely  appropriate  for  trainees  to  get  help  during  early  practice. 
Instructors  or  assistant  instructors  should  be  available  to  provide 
demonstrations  when  necessary,  point  out  cues  and  critical  discriminations, 
and  provide  feedback.  Final  practice,  however,  should  not  contain  anything 
that  will  not  appear  on  the  test  or  on  the  job.  Instructors,  evaluators, 
observers,  and  other  students  should  not  provide  assistance  that  will  not  be 
present  in  the  "real  world."  Extra  hints  given  during  final  practice  will  cause 
more  students  to  fail  the  test  (once  they  don't  have  the  assistance  to  rely 
on  when  taking  the  test).  In  addition  to  obvious  help  during  hands-on 
practice,  such  as  the  instructor  doing  any  part  of  the  task  that  the  trainee  is 
supposed  to  perform,  or  offering  hints  or  suggestions,  there  are  also  more 
subtle  prompts,  such  as  clearing  the  throat,  looking  up  to  "heaven," 
muttering  under  the  breath,  etc.,  when  trainees  start  an  action. 
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USE-RULE 


All  steps  and  branching  decisions 
performed  in  the  correct  order 


USE-PRINCIPLE  Explanation  or  prediction  based  on 

the  principle 

Practice  for  an  LO  cannot  be  counted  as  present  in  the  instruction  unless  it 
is  complete.  If  anything  that  needs  to  be  included  is  omitted,  trainees  may 
be  progressed  in  training  without  having  the  necessary  skills  for  task 
performance.  When,  at  some  later  time,  they  cannot  perform  to  criterion, 
the  blame  usually  falls  on  them.  It  is  not  their  fault. 

4.1.4  Ail  Trainees  Did  Not  Practice  (Question  61). 

All  of  the  trainees  did  not  practice,  where  practice  is  defined  as  in  THE 
PROBLEM,  above.  The  emphasis  here  is  on  the  word  "all."  This  is 
essentially  a  counting  exercise.  If  there  were  10  trainees  at  the  training  site 
and  nine  of  them  practiced,  then  "all"  the  trainees  did  not  practice.  When 
all  the  trainees  did  not  receive  practice,  the  problem  is  usually  with  the 
resources  available  for  training.  If  there  was  not  enough  time,  more  time 
should  be  scheduled  or  additional  training  sites  should  be  made  available.  If 
there  were  not  enough  instructors,  then  more  instructors  are  needed  or  more 
time.  Whatever  the  resource  problem  may  be,  it  has  to  be  corrected  or  "all" 
the  trainees  will  not  get  the  chance  to  be  trained. 

4.1.5  All  Trainees  Did  Not  Meet  Required  Standards  in  Final  Practice 
(Question  62). 

All  of  the  trainees  did  not  practice  the  task  until  unassisted  performance  to 
standard  was  reached.  The  question  is,  "How  many  were  not  required  to  do 
this?"  If  10%  are  not  required  to  demonstrate  proficiency  during  practice, 
then  10%  progressed  in  training  without  having  the  necessary  skills  for  task 
performance.  When,  at  some  later  time,  they  cannot  perform  on  a  level  with 
their  peers,  the  blame  usually  falls  on  them.  It  is  not  their  fault.  If  time 
or  resources  positively  prohibit  all  trainees  from  practicing  until  standards  are 
reached,  then  standards  should  be  relaxed  for  all  the  trainees,  equally.  This 
point  cannot  be  emphasized  too  strongly. 

Final  practice  should  be  the  capstone  for  training  event.  All  of  the 
kmow ledge  taught  by  the  instructors  and  the  skills  learned  through 
demonstration  and  subtask  practice  come  together  in  final  practice.  It  is  that 
point  in  training  where  the  task  statement  from  the  training  objective  is 
addressed  fully.  Final  practice  is  usually  conducted  in  training  but  it  is  not 
always  conducted  well.  In  final  practice,  each  trainee  should  practice  until 
performance  at  or  above  standard  is  reached.  Instructors  should  not  allow 
trainees  to  progress  in  training  until  they  have  met  the  standard  on  at  least 
one  unassisted  trial  (two  or  three  are  better). 


js 

% 


4.2  PRACTICE/DEMONSTRATION:  Practice-Remembering 

THE  PROBLEM.  Practice  items  for  memory  level  objectives  were  inadequate. 
Either  memory  aids  were  not  used  where  required,  or  the  Practice 
Remembering  item  was  not  identical  to  the  test  item.  Practice  questions  are 
included  in  instruction  because  they  (1)  focus  on,  and  call  attention  to, 
information  or  tasks  that  must  be  learned,  and  (2)  inform  the  student  what 
the  test  will  be  like.  For  these  reasons,  Practice  Remembering  items  must 
(1)  test  the  same  information  as  the  test,  and  (2)  have  the  same  format  as 
the  test  item  so  the  student  is  not  "tricked." 

EVIDENCE  for  inadequate  Practice  Remembering  items  will  come  from 
Worksheets  Cl.l  Observation  of  Classroom  Instruction,  and  Cl. 2  Observation 
of  Demonstrations. 

Question  45.  Are  memory  aids  used? 

RECOMMENDATIONS  for  training  program  modification. 

4.2.1  Memory  Aids  Not  Used  (Question  45). 

Memory  aids  (also  called  mnemonics)  were  not  used  as  help  for  memory  level 
objectives.  The  practice  remembering  item  differed  in  content  and/or  format 
from  the  test  item,  resulting  in  confusion  to  the  students.  Practice 
Remembering  items  must  be  the  same  as  the  test  item.  They  must  be  the 
same  format  as  the  test  item.  (See  DEMONSTRATION/PRACTICE:  Practice- 
Test  Consistency.) 

Memory  aids  are  most  often  used  with  Remember-Fact  information,  but  they 
should  also  be  used  for  other  content  types  at  the  Remember  level.  Methods 
of  construction  memory  aids  are  described  in  TAEG  Report  //60,  Use  of 
Mnemonics  in  Training  Materials;  A  Guide  for  Technical  Writers,  by  Braby 
and  Kincaid  (197S). 

Memory  Aids  for  Remember  Fact  Objectives:  Clever  rhymes  and  acronyms 
are  often  used  to  help  learners  remember  facts.  You're  probably  familiar 
with  several.  For  instance,  to  help  remember  the  number  of  days  in  each 
month,  is  the  rhyme  that  begins,  "Thirty  days  hath  September  ...."  To  help 
remember  the  notes  of  the  treble  clef  that  appear  between  the  lines,  we  use 
the  acronym  F.A.C.E.  Whenever  you  use  either  of  these  techniques, 
remember  that  while  clever  rhymes  and  acronyms  are  very  helpful,  contrived 
aids  are  very  confusing.  When  materials  do  not  lend  themselves  to  the 
creation  of  good  rhymes  or  acronyms,  it's  better  to  use  a  different  type  of 
help.  Chunking  is  another  type  of  memory  aid.  Chunking  is  used  to  break 
down  large  memory  tasks  into  smaller  logical  groups. 
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Memory  Aids  for  Remember -Concept,  Rule,  or  Procedure  Objectives.  With 
rules,  concepts,  or  procedures  which  are  difficult  to  remember  but  commonly 
used  on  the  job,  "oid  hands"  at  the  job  may  have  developed  some  verbal 
memory  joggers  to  help  them.  Often  these  memory  aids  help  them  remember 
only  the  most  difficult  step,  operation,  or  characteristic.  For  example, 
aviators  use  this  jingle  to  help  them  remember  whether  to  add  or  subtract 
when  computing  magnetic  variation  based  on  true  variation:  "East  is  least, 
but  West  is  best."  When  teaching  naive  learners  this  rule,  this  jingle  could 
be  presented  and  explained  so  that  they  can  remember  that  when  heading 
East  they  should  subtract,  when  heading  West  they  should  add. 

Verbal  memory  aids  are  often  quite  difficult  to  create  but  if  they  are  readily 
available  can  be  quite  useful.  Another  advantage  of  using  "tried  and  true" 
jingles,  acronyms,  or  other  memory  joggers  is  that  there  is  some  proof  that 
they  have  been  helpful  because  they  have  endured. 


4.3  PRACTICE/DEMONSTRATION:  Practice  Using 


THE  PROBLEM;  Practice  Using  items  were  inadequate.  They  did  not  reflect 
what  is  to  be  done  on  the  job  or  in  later  training,  or  were  not  sequenced 
properly,  or  did  not  differ  from  the  examples. 

EVIDENCE  for  inadequate  Practice  Using  items  will  come  from  Worksheets 
Cl. 3  Observation  of  Insolated  Practice,  and  Cl. 7  Observation  of  Integrated 
Practice. 

Question  47.  Are  PRACTICE  USING  items  sequenced  from  easy  to 
hard? 

Question  48.  Do  PRACTICE-USING  items  provide  opportunities  for 
COMMON  ERRORS  to  be  made? 

Question  49.  Are  PRACTICE  items  different  from  EXAMPLES? 

Question  50.  Does  PRACTICE  USING  integrate  tasks  as  they  are 
integrated  in  the  "real  world?" 

RECOMMENDATIONS  for  training  program  modification. 

4.3.1  Items  Not  Sequenced  From  Easy  to  Hard  (Question  47). 

Practice  Using  items  were  not  sequenced  from  easy  to  hard.  Tasks  are  often 
difficult  to  learn  when  their  full  conditions  and  standards  are  imposed  on 
early  practice  items.  In  order  to  teach  someone  how  to  perform  such  tasks, 
it  is  often  necessary  to  make  it  easier  to  do  at  the  beginning.  To  do  this, 
the  conditions  and  standards  can  be  relaxed.  The  action,  however,  should 
never  be  changed. 

4.3.2  Opportunities  Not  Provided  for  Common  Errors  To  Be  Made  During 
Practice  (Question  48). 


Practice  questions  or  exercises  did  not  allow  students  to  choose  or  commit 
most  errors  common  in  the  real  job  task.  Practice  should  include  the 
opportunity  for  students  to  make  common  errors.  The  feedback  should  then 
clearly  show  why  the  error  is  wrong,  and  how  to  avoid  it. 

Identification  of  common  errors  often  requires  consultation  with  a  subject 
matter  expert.  His  familiarity  with  the  task  or  objective  should  enable  him  to 
think  of  the  common  mistakes  trainees  make  when  performing  the  task  or  the 
behavior  required  by  the  objective.  If,  for  example,  the  objective  involves 
applying  a  rule,  common  mistakes  might  involve  things  like: 

1.  Confusing  one  type  of  data  with  another  (e.g.,  mixing  similar 
symbols,  "C"  for  "Cl"  in  chemical  formulas). 

2.  Confusing  the  operations  involved  in  using  the  rule  (e.g.,  adding 
instead  of  subtracting). 

3.  Misreading  tables  or  other  presentations  of  the  data  (e.g.,  taking 
information  from  the  wrong  column  or  row  in  a  table,  etc.). 


4. 


Being  unable  to  use  difficult  forms  of  data  (e.g.,  numbers  in 
fraction,  large  decimal,  or  square  root  form). 


5.  Making  computational  errors  (e.g.,  misplacing  decimals  or  making 
common  multiplication  errors). 

4.3.3  Practice  Items  Not  Different  Than  Examples  (Question  49). 

Practice  items  for  Use  Concept,  Rule,  or  Principle  objectives  did  not 
differ  from  previous  examples.  Practice  Using  items  for  concepts,  rules,  and 
principles  must  be  different  from  the  test  items  and  examples  (even  though 
they  are  the  same  format).  They  must  be  different  because  these  content 
types  require  the  student  to  be  able  to  deal  with  problems  or  situations  that 
are  new.  (For  procedures,  nothing  new  is  required,  so  the  practice  is  the 
same  as  the  test.)  One  exception  is  that  common  error  items  might  be  the 
same. 


4.3.4  Practice  Requirements  Differ  from  Job  Requirements  (Question  50). 

Practice  Using  did  not  integrate  tasks  as  they  are  integrated  in  the  "real 
world."  The  practice  session  should,  as  closely  as  possible,  match  actual  job 
conditions  and  should  require  trainees  to  practice  tasks  together  that  would  be 
performed  together  on  the  job.  In  other  words  the  practice  should 
approximate  job  requirements.  The  extent  to  which  this  can  be  done  may  be 
limited  by  resource  constraints,  but  such  constraints  can  often  be  overcome 
with  a  little  ingenuity.  Talk  to  the  training  observers  to  find  out  why  they 
rated  Question  50  a  "2"  or  "3."  Take  their  observations  to  the  training 
developer  to  see  if  more  realistic  practice  can  be  developed. 


4.4  PRACTICE/DEMONSTRATION:  Demonstrations 

THE  PROBLEM.  The  demonstration  that  was  given  to  the  trainees  was  not 
effective.  The  instructor  did  not  show  how  to  correct/avoid  common  errors, 
the  steps  in  the  demonstration  were  too  large  or  small,  or  they  were  not 
demonstrated  in  the  same  sequence  as  they  were  performed  in  the  real  world. 


EVIDENCE  for  poor  demonstrations  will  come  from  the  Worksheets  Cl.l 
Observation  of  Classroom  Instruction,  and  C1.2  Observation  of 
Demonstrations. 

Question  42.  Do  DEMONSTRATIONS  show  how  to  correct/avoid  common 
errors? 

Question  43.  Are  steps  in  a  DEMONSTRATION  the  appropriate  size? 

Question  44.  Are  tasks  and  task  steps  DEMONSTRATED  in  the  same 
sequence  as  they  are  performed  in  the  "real  world?" 

RECOMMENDATIONS  for  training  program  modification. 

4.4.1  Demonstrations  Did  Not  Show  How  to  Correct  or  Avoid  Common  Errors 
(Question  42). 

The  instructor  did  not  include  guidance  on  methods  for  avoiding  or  correcting 
errors  typically  made  on  the  job.  Demonstrations  should  include  an 
explanation  of  the  steps  on  which  students  usually  make  errors  and,  if 
possible,  tell  the  student  how  the  error  can  be  avoided.  You  may  need  to 
consult  with  a  subject  matter  expert  to  determine  the  likelihood  of  common 
errors. 

4.4.2  Demonstrations  Not  Conducted  in  Small  Enough  Steps  (Question  43). 

Demonstrations  were  not  conducted  in  small  enough  steps  so  that  the  trainees 
could  easily  follow  the  instructor's  actions.  Step  size  refers  to  the  amount  of 
information  needed  to  get  from  one  step  to  another  in  task  performance.  We 
are  all  familiar  with  very  large  step  sizes  from  high  school.  They  usually 
came  in  mathematics  books  when  the  author  said,  "The  solution  is  obvious  and 
left  up  to  the  student."  The  solution  was  seldom  obvious  because  the  step 
size  was  too  big.  There  was  too  much  information  needed  to  get  from  that 
to  problem  solution.  Large  step  sizes  are  all  right  for  trainees  who  have 
already  mastered  the  tasks  but  trainees  new  to  the  task  have  to  have  it 
broken  down  into  small  steps  so  they  can  follow  along  without  getting  lost 
between  steps. 

Clues  to  problems  with  step  size  will  come  from  observations  made  by  the 
training  observer.  If  instructors  backtracked  to  add  information  (filling  in 
between  steps)  that  they  felt  was  necessary  for  complete  understanding,  or 
added  information  to  the  instruction  that  is  not  part  of  the  current  lesson 
materials,  then  they  are  already  aware  of  the  step  size  problem.  They  need 
to  bring  the  lesson  plan  into  conformance  with  practice  by  adding  the 
additional  steps  or  information  to  the  plan  for  giving  the  demonstration. 
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If  instructors  had  to  backtrack  to  add  information  in  response  to  trainee 
questions,  and  this  information  appeared  to  add  to  trainee  understanding,  the 
instructors  have  not  identified  a  set  of  needed  additional  steps.  They  need  to 
organize  the  information  or  steps  they  developed  in  response  to  questions  and 
include  it  in  the  lesson  plan. 

4.4.3  Tasks  or  Task  Steps  Demonstrated  Out  of  Sequence  (Question  44). 

The  instructor  did  not  demonstrate  tasks  or  task  steps  in  the  same  sequence 
as  they  are  performed  in  the  real  world.  It  is  crucial  that  the  instructor 
deal  with  exactly  the  same  steps  in  the  same  order  as  presented  in  the  lesson 
plan  or  technical  manual  to  avoid  confusing  the  student.  On  rare  occasions, 
the  order  of  certain  steps  within  the  procedure  may  be  optional  and  should 
then  be  indicated  as  such. 


4.5  PRACTICE/DEMONSTRATION:  Team  Practice 


THE  PROBLEM.  Team  practice,  defined  as  the  practice  of  a  team  function 
by  all  crew/team  members  in  a  realistic  setting,  either  did  not  happen  or  did 
not  happen  as  it  should  have.  Team  practice  is  essential  to  the  development 
of  team  proficiency,  i.e.,  the  ability  of  a  team  or  crew  to  successfully 
perform  the  coordination  and  communication  tasks  required  of  the  team  in 
addition  to  the  individual  skills  involved.  No  matter  how  proficient  individual 
team  members  are  in  their  areas  of  individual  responsibility,  they  will  not 
learn  to  function  as  a  team  unless  they  practice  as  a  team.  It  is  essential 
that  team  practice  conditions  be  the  same  as  those  of  the  real  task  and  that 
adequate  team  feedback  be  provided. 

EVIDENCE  for  missing  or  poor  team  practice  will  come  from  Worksheeet  Cl. 7 
Observation  of  Integrated  Practice. 

Question  65.  Is  TEAM  PRACTICE  provided? 

Question  66.  Are  TEAM  PRACTICE  CONDITIONS  the  same  as  (or  as 
close  as  possible  to)  those  of  the  real  task? 

Question  67.  Is  TEAM  PRACTICE  FEEDBACK  provided? 

Question  68.  Is  FEEDBACK  HELP  for  TEAM  PRACTICE  adequate? 

RECOMMENDATIONS  for  training  program  modification. 

4.5.1  Team  Practice  Missing  (Question  65). 

Trainees  had  no 'opportunity  to  practice  team  skills.  The  purpose  of  team 
practice  is  to  provide  for  the  acquisition  and/or  enhancement  of  team  skills. 
Practice  to  proficiency  on  individual  tasks  will  not  produce  individuals  skillful 
in  team  situations.  The  development  of  some  sort  of  "team  awareness,"  or 
"team  quality,"  is  necessary.  Individual  proficiency  is  important  to  the  overall 
success  of  the  team,  but  it  does  not  outweigh  such  team  skill  as  cooperation, 
crew  coordination,  communication,  and  tactical  decision  making. 

In  order  to  present  trainees  with  the  opportunity  to  practice  the  variety  of 
team  interactions  present  in  tactical  combat  situations,  it  is  necessary  to 
develop  a  progression  of  practice  scenario  presentations.  This  series  of 
scenarios  should  incorporate  increasing  complexity  in  regard  to  team  member 
roles,  information  flows,  decision  and  problem  solving  requirements, 
coordination  activities,  and  evolution  of  tactics  as  the  battle  progresses. 

4.5.2  Team  Practice  Conditions  Unrealistic  (Question  66). 

Team  practice  conditions  were  not  the  same  as  (or  as  close  as  possible  to) 
those  of  the  real  tasks.  In  order  to  adequately  prepare  trainees  for  combat, 
team  practice  scenarios  should  be  as  realistic  as  possible.  Initial  team 
practice  conditions  may  be  simplified  so  as  not  to  overwhelm  the  trainees. 
Problem  complexity  should  be  increased  as  trainees  demonstrate  proficiency  in 
their  established  team  roles.  For  example,  early  scenarios  might  deal  with 


established  situations  in  which  all  action-relevant  environmental  conditions  and 
states  of  the  system  are  specifiable  and  predictable.  Later  team  practice 
might  involve  situations  in  which  the  previously  predictable  or  knowledgeable 
conditions  or  states  no  longer  hold.  An  example  would  be  the  sudden 
introduction  of  a  disabled  truck,  or  the  failure  of  expected  personnel  to 
arrive.  The  latter  kind  of  team  practice  more  closely  approximates  combat 
conditions. 


Trainees  were  not  told  whether  their  team  practice  mission  was  successful  or 
not,  and  if  not,  what  they  did  wrong.  Performance  feedback  is  unquestionably 
the  single  most  critical  component  in  team  practice.  It  is  essential  that 
team  practice  include  clear  and  immediate  reinforcement  following  each 
correct  team  response.  Practice  in  the  absence  of  team  reinforcement  for 
criterion-level  performance  may  well  lead  to  a  decrease  in  team  proficiency. 


An  approach  used  to  provide  team  members  with  feedback  or  "knowledge  of 
results"  is  the  post-exercise  debriefing.  Debriefing  sessions  following  team 
practice  sessions  provide  an  opportunity  for  the  team  to  examine  individual 
proficiencies  and  to  explore  alternative  ways  of  organizing  the  task  in  order 
to  develop  more  efficient  and  proficient  team  performance. 


Feedback  help  for  team  practice  was  confusing  or  did  not  provide  sufficient 
explanation.  In  order  for  team  members  to  learn  from  their  mistakes,  they 
must  be  told  how  to  correct  them.  However,  a  major  source  of  difficulty  in 
team  training  is' the  identification  and  correction  of  individual  errors.  In 
many  complex  team  tasks,  there  are  often  many  possible  correct  procedures, 
making  error  identification  difficult.  These  conditions  may  foster  the  problem 
of  monitoring  or  supervising  team  responses  to  evaluate  their  adequacy. 
Practice  scenarios  should  be  analyzed  to  determine  critical  procedures, 
decision  points,  communications,  and  coordinated  activities  which  are  linked  to 
the  mission  outcome.  This  process,  although  subjective  in  nature,  might  be 
accomplished  by  a  consensus  of  knowledgeable  instructors.  A  feedback 
schedule  may  then  be  established  for  critical  mission  events. 


4.6  PRACTICE/DEMONSTRATION:  Feedback 


THE  PROBLEM.  Trainees  were  not  told  whether  they  were  right  or  wrong  on 
each  practice  item,  and,  if  wrong,  the  correct  answer  and  why.  For 
performance  tasks  on  which  a  step-by-step  process  is  required,  feedback 
should  be  given  for  each  step. 

EVIDENCE  for  poor  or  missing  feedback  during  practice  will  come  from  the 
following  worksheets: 

Cl.l  Observation  of  Classroom  Instruction 

Cl. 3  Observation  of  Isolated  Practice 

C1.7  Observation  of  Integrated  Practice 

Question  63.  Is  FEEDBACK  provided  for  PRACTICE? 

Question  64.  Is  FEEDBACK  Help  adequate? 

4.6.1  Feedback  Not  Provided  During  Practice  (Question  63). 

Trainees  were  not  given  feedback  during  practice,  as  soon  as  possible 
following  their  actions.  Trainees  learn  what  they  practice.  It  is  important 
that  they  practice  correct  performance  at  all  times.  All  trainee  performance 
should  be  monitored  by  an  instructor  to  insure  that  trainees  are  practicing 
correct  performance.  Since  trainees  may  have  no  clear  memory  a  few 
seconds  later  of  how  they  responded,  trainee  errors  should  be  corrected  as 
soon  after  they  occur  as  possible.  Encouragement  should  be  given  during 
error-free  performance  to  reinforce  and  "lock  in"  this  performance.  This 
necessary  interaction  between  the  trainee  and  the  instructor  must  take  place 
while  practice  is  going  on.  Post-practice  critiques  will  not  correct  faulty 
performance  and  do  not  "lock  in"  correct  performance  very  effectively. 

Practice  feedback  for  REMEMBER  level  objectives  simply  provides  the  student 
with  a  correct  answer  to  compare  with  his  own.  Adequate  practice  feedback 
should  provide  a  complete  answer  to  the  item,  provide  the  learner  with 
immediate  access  to  the  feedback,  and  clearly  display  the  feedback  and 
identify  the  items  to  which  it  relates. 

Practice  feedback  for  USE  CONCEPT  objectives  should  contain  the  correct 
and  clearly  identified  answer  to  the  problem.  It  should  isolate  critical 
characteristics,  and  point  out  the  missing  critical  characteristics  in  non¬ 
examples.  To  the  learner  it  is  quite  important  to  be  shown  why  a  practice 
item  is  an  example  of  a  concept  or  why  it  is  not.  USE  CONCEPT  practice 
feedback  should  also  point  out  and  explain  common  errors  and  give  a 
reference  for  correcting  them,  when  appropriate. 

Practice  feedback  for  USE  PROCEDURE  objectives  involves  giving  the  learner 
detailed  information  on  the  adequacy  of  his  performance.  The  instructor 
should  respond  to  each  mistake  the  student  makes.  Since  the  purpose  of 
feedback  is  to  help  the  student  learn  the  material,  whenever  possible  the 


instructor  should  have  the  learner  discover  the  reason  for  his  mistake.  He 
should  direct  the  learner  to  a  reference  where  he  can  locate  the  source  of 
his  problem.  Again,  while  the  instructor  should  give  feedback  on  both  correct 
and  incorrect  responses,  he  should  emphasize  the  correction  of  errors. 

Practice  feedback  for  USE  RULE  objectives  does  not  merely  provide  the 
student  with  the  correct  answer.  The  instructor  should  add  information  which 
explains  how  to  arrive  at  the  correct  answer  so  that  the  learner  is  able  to 
find  mistakes.  Adequate  feedback  for  USE  RULE  practice  items  should 
contain  the  correct  and  clearly  identified  answer  to  the  problem.  It  should 
present  a  step-by-step  application  of  the  rule.  When  formulas  are  used, 
feedback  should  consist  of  showing  all  the  logical  steps  worked  out.  Finally 
the  feedback  should  point  out  and  explain  common  errors,  and  give  a 
reference  for  correcting  them,  when  appropriate. 

4.6.2  Feedback  Help  Inadequate  (Question  64). 

Feedback  Help  was  confusing  or  provided  insufficient  explanation,  e.g., 
consisted  of  a  repetition  of  the  original  presentation.  Students  sometimes 
make  errors  because  they  did  not  learn  from  the  original  presentation. 
Therefore,  when  possible,  feedback  should  include  additional  information  or  a 
different  version  of  the  information  than  was  originally  presented.  Some 
types  of  help  for  different  content  types  are  given  in  the  Recommendations 
for  Question  35  paragraph  4.4.4),  "Does  Statement  Help  Provide  Sufficient 
Explanation?" 
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4.7  PRACTICE/DEMONSTRATION:  Job  Performance  Aids 


THE  PROBLEM.  Job  performance  aids  were  not  an  integral  part  of  training. 

If  job  performance  aids  exist  for  a  task,  they  should  be  used  actively  during 
training,  and  should  be  usable.  Training  should  focus  more  on  the  use  of  the 
job  performance  aid  than  on  task  performance  without  benefit  of  the  job 
performance  aid. 

EVIDENCE  for  job  performance  aids  not  being  an  integral  part  of  training  will 
come  from  the  following  worksheets: 

Cl. I  Observation  of  Classroom  Instructon 

C1.2  Observation  of  Demonstrations 

C1.3  Observation  of  Isolated  Practice 

C1.7  Observation  of  Integrated  Practice 

Question  36.  Does  training  include  instruction  on  the  use  of  required  job 
performance  aids? 

Question  51.  Are  job  performance  aids  (JPAs)  usable** 

Question  52.  Do  all  students  use  the  job  performance  aid  (JPA)? 
RECOMMENDATIONS  for  training  program  modification. 

4.7.1  Trainees  Not  Instructed  on  the  Use  of  Job  Performance  Aids  (Question 

W. 

Job  performance  aids  were  not  adequately  introduced  and  explained; 
consequently,  students  did  not  know  how  to  use  them.  Instruction  should 
include  objectives  on  how  to  use  the  aids,  or  how  to  find  them,  or  additional 
fact  objectives  on  the  vocabulary.  The  job  performance  aids  should  be  used 
by  the  instructor  in  examples  and  demonstrations. 

4.7.2  Job  Performance  Aids  Not  Usable  (Question  51). 

Job  performance  aids  were  inaccurate  and/or  were  not  of  much  help  to  the 
trainees.  There  are  n  any  poorly  designed  job  performance  aids  in  the  world 
of  work.  Job  performance  aids  may  have  been  hard  to  find,  or  hard  to  use, 
or  may  have  used  different  technical  vocabulary  from  that  used  in  training. 

If  this  happened,  it  is  obvious  that  new  job  performance  aids  are  needed. 

Until  these  job  performance  aids  are  produced,  training  is  probably  better  off 
without  them.  (In  that  event,  the  objective  should  be  rewritten  as  USE 
UNAIDED,  and  a  previous  REMEMBER  objective  should  be  included  in  the 
instruction.) 


Job  performance  aids  were  available/required  for  task  performance,  but  some 
or  most  of  the  trainees  did  not  use  them.  The  job  performance  aids  may 
have  been  hard  to  find,  or  hard  to  use,  but  in  any  case,  they  were  not  an 
integral  part  of  instruction.  If  a  job  performance  aid  for  this  task  currently 
exists  but  was  not  made  an  integral  part  of  training,  the  training  program 
should  be  revamped  to  focus  on  this  job  performance  aid.  The  training 
process  should  be  such  that  the  primary  emphasis  is  placed  on  the  job 
performance  aid  rather  than  task  completion.  After  the  initial  terms  and 
concepts  needed  to  use  the  job  performance  aid  to  perform  the  task  are 
mastered  by  the  trainees,  the  demonstration  and  practice  events  should  aim 
toward  making  the  trainees  masters  of  the  job  performance  aid.  The  test, 
too,  should  be  on  using  the  job  performance  aid  to  perform  the  task.  The 
most  useful  examples  here  are  in  the  maintenance  area.  Maintenance 
personnel  should  be  taught  to  use  the  manuals  and  associated  tools  to 
repair/replace  equipment  rather  than  being  taught  to  perform  the  repair/ 
replace  functions  only.  This  requires  that  each  trainee  be  given  a  job  aid  at 
the  beginning  of  training  to  keep  and  to  annotate  as  deemed  necessary  by  the 
trainee. 
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4.8  PRACTICE/DEMONSTRATION:  Practice-Test  Consistency 


THE  PROBLEM.  Practice  items  did  not  match  the  task/content  level, 
conditions,  or  standards  of  the  test  item,  or  the  practice  item  format  was  not 
the  same  as  the  test  item  format.  Practice  questions  or  exercises  are 
included  in  instruction  because  they  (1)  focus  on,  and  call  attention  to, 
information  or  tasks  that  must  be  learned;  and  (2)  inform  the  student  about 
what  the  test  will  be  like.  For  these  reasons,  practice  items  should  match 
the  test  items. 

EVIDENCE  for  a  practice-test  mismatch  will  come  from  Worksheets  Cl. 3 
Observation  of  Isolated  Practice,  and  Cl. 7  Observation  of  Integrated 
Practice. 

Question  46.  Does  each  PRACTICE  REMEMBERING  item  have  the  same 
content  and  format  as  the  test  item? 

Question  53.  Does  the  TASK  LEVEL  of  the  PRACTICE  item  match  that 
of  the  test  item(s)? 

Question  54.  Does  the  CONTENT  TYPE  of  the  PRACTICE  item  match 
that  of  the  test  item(s)? 

Question  55.  Does  the  FORMAT  of  the  PRACTICE  item  match  that  of 
the  test  item? 

Question  56.  Do  the  CONDITIONS  of  each  FINAL  PRACTICE  item 
match  those  of  the  test  item(s)? 

s 

Question  57.  Do  the  STANDARDS  of  each  FINAL  PRACTICE  item 
match  those  of  the  test  item(s)? 

RECOMMENDATIONS  for  training  program  modification. 

4.8.1  Practice  Item  Task/Content  Level  Improperly  Matched  with  Task  Item 


Task/Content  Level  (Questions  46,  53  and  54 


The  task/content  level  of  the  practice  item  did  not  match  the  task/content 
level  of  the  test  item.  Recalling  the  task/content  matrix  (Figure  2),  this 
means  that  the  action  verb  on  the  practice  item  was  not  the  same  or  the 
action  verb  on  the  test  item,  or  that  the  same  behavior  was  not  required. 

The  result  is  that  the  students  are  practicing  something  different  from  what 
they  will  later  be  tested  on. 

Here  are  some  examples  of  practice  item-test  item  mismatches  on 
task/content  level: 

(1)  PRACTICE  ITEM:  In  the  space  below,  list  the  major  battles  in  France 
during  World  War  II  (as  outlined  in  your  text). 

TEST  ITEM:  In  the  space  below,  list  the  major  battles  in  France 
during  World  War  II  (as  outlined  in  your  text).  Include  the  dates  of 
each  battle. 
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TASK  LEVEL 


CONTENT  TYPE 


FACT 

CONCEPT 

PROCEDURE 

RULE 

PRINCIPLE 

REMEMBER 

RECALL  OR  RE¬ 
COGNIZE  NAMES. 
PARTS,  DATES, 
PLACES,  VO¬ 
CABULARY  DEF¬ 
INITIONS,  ETC. 

REMEMBER  THE 
CHARACTERISTICS 
OF  EACH  CON¬ 
CEPT  AND  THE 
GUIDELINES  FOR 
CLASSIFICATION. 

REMEMBER  THE 
STEPS  OF  THE 
PROCEDURE. 

REMEMBER  THE 
FORMULA  OR 

THE  STEPS  OF 

THE  RULE. 

REMEMBER  THE 
RELATED  PROCE¬ 
DURES.  OR  CON¬ 
CEPTS  TO  BE 
SELECTED  FROM 
AND  APPLIED  OR 
THE  STATEMENT 

OF  THE  PRINCIPLE. 

USE -UNAIDED 

CLASSIFY  OR 
CATEGORIZE 
OBJECTS.  EVENTS, 
IDEAS',  ACCORDING 
TO  THEIR  CHARAC¬ 
TERISTICS.  WITH 

NO  MEMORY  AID. 

APPLY  THE 

STEPS  OF  THE 
PROCEDURE  IN 

A  SINGLE  SIT¬ 
UATION  OR  ON 

A  SINGLE  PIECE 
OF  EQUIPMENT. 
WITH  NO  MEM¬ 
ORY  AID. 

APPLY  THE 
FORMULA  OR 
RULE  TO  A 
VARIETY  OF 
PROBLEMS  OR 
SITUATIONS. 

WITH  NO  MEM¬ 
ORY  AID. 

USE  THE  PRIN¬ 
CIPLE  TO  DES¬ 
CRIBE  WHY  OR 

HOW  THINGS 
HAPPEN  OR  WHAT 
WILL  HAPPEN. 

WITH  NO  MEMORY 
AID. 

USE -AIDED 

GIVEN  CONCEPT 
CHARACTERIS¬ 
TICS  AND  GUIDE¬ 
LINES.  CATE¬ 
GORIZE  OBJECTS, 
EVENTS.  IDEAS. 
ACCORDING  TO 
CHARACTERIS¬ 
TICS. 

GIVEN  STEPS  OF 
THE  PROCEDURE. 
APPLY  THE  PRO¬ 
CEDURE  IN  A 
SINGLE  SIT¬ 
UATION.  OR 

ON  A  SINGLE 

PIECE  OF 
EQUIPMENT. 

GIVEN  THE 
FORMULA  OR 
RULE  STEPS. 
APPLY  THE 
FORMULA  OR 
RULE  TO  A 
VARIETY  OF 
PROBLEMS  OR 
SITUATIONS. 

GIVEN  A  STATE¬ 
MENT  OF  THE 
PRINCIPLE, 
DESCRIBE  WHY 

OR  HOW  THINGS 
HAPPEN  OR 

WHAT  WILL  HAP¬ 
PEN. 

Figure  2  TASK/CONTENT  CLASSIFICATION  MATRIX 
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COMMENT:  Nothing  about  dates  is  mentioned  in  the  practice  item. 
Although  the  action  verb  is  the  same,  the  behavior  called  for  on  the 
test  is  different. 


(2)  PRACTICE  ITEM:  In  the  space  below,  write  the  steps  of  the  procedure 
for  message  recepton  and  duplicate  checking.  You  will  receive  full 
credit  only  if  you  list  all  steps  exactly  as  stated  in  NTP-4  in  the 
correct  order. 

TEST  ITEM:  Use  the  current  edition  of  NTP-4  to  receive  the  following 
message  and  check  for  duplicates. 

COMMENT:  The  practice  item  is  Remember-Procedure.  The  test  item 
is  Use- Aided  Procedure. 


(3)  PRACTICE  ITEM:  In  the  space  below  write  the  principles  of  operation 
of  a  rotary  gear  pump,  as  they  were  described  in  TM-XXX. 

TEST  ITEM:  In  the  space  below  list  the  part  names  and  functions  of  a 
rotary  gear  pump. 

COMMENT:  The  task/content  level  of  the  practice  items  is  Remember- 
Principle,  while  the  task/content  level  of  the  test  item  is  Remember- 
Fact. 

If  the  practice  item  and  test  item  differ  in  task/content  level,  check  the 
objective  to  see  which  is  correct  and  recommend  that  the  training  or  test 
developer  modify  the  incorrect  item. 

4.8.2  Practice  Item  Format  Improperly  Matched  with  Test  Item  Format 
(Question  55). 

The  practice  item  format  was  not  the  same  as  the  test  item  format.  There 
are  a  number  of  different  item  formats.  These  are  listed  below: 

•  True-False 

•  Multiple  Choice 

•  Matching 

•  Fill-In 

•  Short  Answer 

•  Listing 

•  Performance 

Here  are  two  examples  of  practice  item-test  item  mismatches  on  item 
format: 

(1)  PRACTICE  ITEM:  Which  of  the  following  is  the  correct  rule  for 
finding  total  inductance  in  a  series  circuit? 


B.  \c  =  2tt*RLc 


C.  R  =  IE 

D.  None  of  the  above. 

TEST  ITEM:  In  the  space  below  write  the  correct  rule  for  finding 
total  inductance  in  a  series  circuit. 

COMMENT:  The  practice  item  is  multiple  choice  and  calls  for  a 
selected  response.  The  test  item  calls  for  a  constructed  response. 

Since  it  is  easier  to  select  the  correct  response  than  to  provide  it  from 
memory,  the  students  have  not  had  sufficient  practice  (assuming  the 
test  item  matches  the  objective). 


(2)  PRACTICE  ITEM:  Using  your  scope  and  audio  output,  identify  the 
following  signals  according  to  circular,  sector,  conical,  or  steady  types 
of  scans. 

TEST  ITEM:  Describe  the  radar  signal  parameter  measurement 
characteristics  of  each  type  of  scan:  circular,  sector,  conical,  and 
steady.  For  credit  all  characteristics  must  be  described. 

COMMENT:  The  practice  item  is  a  performance  item,  while  the  test 

item  has  a  fill-in  format.  If  the  objective  calls  for  a  paper  and  pencil 

answer,  the  practice  provided  may  be  "overkill." 

% 

If  the  practice  item  and  test  item  differ  in  format,  check  the  objective 
to  see  which  is  correct  and  recommend  that  the  training  or  test  developer 
modify  whichever  item  is  incorrectly  formatted. 

4.8.3  Final  Practice  Conditions  Improperly  Matched  with  Test  Conditions 
(Question  56). 

The  conditions  in  the  final  practice  item,  or  the  conditions  under  which 
practice  is  administered,  did  not  match  the  conditions  in  the  test  item.  If 
the  conditions  do  not  match,  students  may  not  have  been  adequately  prepared 
for  the  test.  Here  are  some  examples  of  practice  test  items  with 
mismatched  conditions. 

(1)  PRACTICE  ITEM:  Which  of  the  following  is  the  correct  rule  for 
finding  total  inductance  in  a  series  circuit? 

A.  E  =  IR 

B.  Xc  =  2ffRLc 

C.  R  =  IE 

D.  None  of  the  above. 

TEST  ITEM:  In  the  space  below,  write  the  correct  rule  for  finding 
total  inductance  in  a  series  circuit. 


COMMENT:  In  the  practice  item,  the  student  is  given  several  rules, 
one  of  which  might  be  the  right  one.  In  the  test  item,  the  student  is 
not  given  anything  except  instructions  to  state  the  rule. 


(2)  PRACTICE  ITEM:  _ is  the  symbol  for  resistor. 

TEST  ITEM:  Several  schematic  symbols  for  common  electronic 
components  are  shown  below.  Write  the  name  of  the  component 
represented  by  each  symbol  in  the  space  provided. 

COMMENT:  The  test  item  says  that  the  student  is  given  symbols. 

The  practice  item  does  not  give  a  symbol. 

(3)  PRACTICE  ITEM:  List  the  steps  for  calibratng  an  10XMX-2817V  test 
prod  using  an  oscilloscope.  For  full  credit  you  must  complete  this  in 
ten  minutes.  You  will  receive  partial  credit  if  you  take  longer. 

COMMENT:  The  test  item  gives  a  student  an  oscilloscope  to  use.  The 
practice  item  requires  a  paper  and  pencil  answer. 

(4)  PRACTICE  ITEM:  Here  are  diagrams  representing  the  last  two  phases 
of  the  basic  steam  cycle.  Using  these  as  memory  aids  for  the 
principles  illustrated  by  them,  analyze  how  the  phases  apply  in  the 
correct  operation  of  the  auxiliary  condensate-system. 

TEST  ITEM:  Here  are  diagrams  representing  the  phases  of  the  basic 
steam  cycle.  Using  these  as  memory  aids  for  the  principles  illustrated 
by  them,  analyze  how  the  phases  apply  in  the  correct  operation  of  the 
auxiliary  condensate  system. 

COMMENT:  The  test  item  states  that  diagrams  are  given  for  the 
entire  steam  cycle.  The  practice  item  specifies  that  only  the  diagrams 
for  the  last  two  phases  are  given. 

If  the  practice  and  test  items  differ  in  condition,  check  the  objective  to  see 
which  is  correct  and  recommend  that  the  training  or  test  developer  modify 
the  incorrect  item. 

^•8.4  Final  Practice  Improperly  Matched  with  Test  Standards  (Question  57). 

The  standards  in  the  practice  item,  or  the  standards  for  scoring  the  practice 
item,  did  not  match  the  standards  in  the  test  item.  In  criterion  referenced 
instruction,  standards  are  not  arbitrarily  selected.  It  makes  no  sense,  for 
example,  to  require  a  student  to  get  80%  of  the  items  right  during  practice, 
if  he  needs  to  recall  all  of  the  information  on  the  test.  The  standard 
specified  in  the  objective  should  be  used  for  both  final  practice  and  the  test. 
Here  are  two  examples  of  practice  and  test  items  with  inconsistent  standards. 


(1)  PRACTICE  ITEM:  In  the  space  below  list  the  part  names  and  functions 
of  a  rotary  gear  pump. 


TEST  ITEM:  In  the  space  below  write  the  principles  of  operation  of  a 
rotary  gear  pump,  as  they  were  described  in  TM-XXX. 

COMMENT:  The  standards  for  the  test  item  are  provided  in  TM-XXX. 
It  is  not  clear  whether  this  manual  has  the  information  required  by  the 
practice  item. 


(2)  PRACTICE  ITEM:  The  following  message  will  be  transmitted  via 

teletype  tape.  Accept  it,  verify  it,  and  log  it.  You  will  receive  full 
credit  if  you  perform  these  steps  accurately  within  30  seconds. 

TEST  ITEM:  The  following  message  will  be  transmitted  via  teletype 
tape.  Process  it  according  to  the  procedure  you  have  learned.  You 
will  receive  full  credit  if  you  perform  the  steps  accurately  within  5 
minutes. 

COMMENT:  The  practice  item  says  30  seconds  per  message.  The  test 
item  allows  5  minutes. 

If  the  standards  of  final  practice  do  not  match  those  of  the  test,  the  practice 
and/or  test  standards  should  be  brought  in  line  with  the  objectives.  If  the 
objectives  are  wrong,  they  should  be  changed. 


5. 


TESTING 


5.1  TESTING:  Objectives-Test  Consistency 

THE  PROBLEM.  The  test  was  not  a  good  measure  of  the  training  objective. 
That  is,  it  was  not  a  good  measure  of  the  extent  to  which  the  training 
program  provided  trainees  with  the  skills  and  knowledges  necessary  to  perform 
the  required  tasks  to  standard.  The  purpose  of  a  training  program  is  to  bring 
everyone  to  proficiency  on  the  tasks  in  the  training  objectives.  The  purpose 
of  the  test  is  to  determine  if  the  trainees  are  actually  proficient  on  the  tasks 
in  the  training  objectives.  If  the  test  tasks,  conditions,  and  standards  do  not 
match  the  training  tasks,  conditions,  and  standards,  there  is  no  way  of 
knowing  if  the  training  program  has  actually  brought  the  trainees  to 
proficiency. 

EVIDENCE  for  a  test-training  mismatch  will  come  from  the  following 
worksheets: 

E2.1  Summary  of  Test  Adequacy  Data  for  Objectives 

E2.2  Summary  of  Test  Adequacy  Data  for  the  Test 

Question  1.  Is  the  test  FORMAT  appropriate  to  the  TASK  LEVEL  and 
CONTENT  TYPE  of  the  OBJECTIVE? 

Question  2.  Are  there  test  items  for  the  TLO  or  all  of  its  critical 
parts/LOs? 

Question  3.  Is  there  a  test  item  for  each  critical  part  of  each  LO9 

Question  4.  Does  the  TASK  LEVEL  of  the  test  item  match  the  TASK 
LEVEL  of  its  OBJECTIVE? 

Question  5.  Does  the  content  of  the  test  item  match  the  content  of 
its  OBJECTIVE? 

Question  6.  Do  the  CONDITIONS  of  the  test  item  match  the 
CONDITIONS  of  its  OBJECTIVE? 

Question  7.  Do  the  standards  of  the  test  item  match  the  standards  of 
its  OBJECTIVE? 

RECOMMENDATIONS  for  training  program  modification. 

5.1.1  Test  Format  Inappropriate  To  the  Task/Content  Level  of  the  Objective 


There  are  a  number  of  different  test  item  formats,  and  these  may  be  more 
or  less  appropriate  depending  on  the  task/content  level  of  the  objective. 
Figure  3  shows  the  acceptable  test  formats  for  each  task/content  level. 

Notice  that  at  the  REMEMBER  level,  recognition  items  (multiple-choice, 
matching,  true-false)  are  usually  not  appropriate.  This  is  because  they  don't 
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Figure  3  APPROPRIATE  TEST  FORMATS  FOR  TASK  LEVELS  AND  CONTENT  TYPES 
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test  recall,  only  recognition.  Most  REMEMBER  level  objectives  require  recall 
because  of  the  nature  of  the  job. 

Recognition  can  be  appropriate,  however,  in  two  situations  at  the  REMEMBER 
level.  The  first  occurs  with  FACTS,  when  a  selection  must  be  made  from  a 
group  of  objects,  locations,  etc.  For  example,  the  task  "go  to  the  tool  box 
and  get  a  ball-peen  hammer"  is  a  RE  MEMBER-FACT  recognition  task. 

The  second  situation  can  occur  for  any  content  type  at  the  REMEMBER  level 
when  the  job  only  requires  the  student  to  be  generally  familiar  with  the 
REMEMBER  level  information.  This  situation  only  happens  when  the  student 
is  being  prepared  for  later  on-the-job  or  formal  training,  and  even  then,  only 
when  the  student  will  be  closely  supervised.  This  is  because  the  supervisor 
can  correct  memory  failure.  For  example,  a  student  performing  the  steps  of 
a  maintenance  procedure  on  a  piece  of  equipment  may  not  need  to  have 
memorized  the  steps  if  the  supervisor  was  available  to  correct  any  errors  or 
to  tell  what  to  do  next.  Even  in  this  situation,  it  would  have  been  more 
efficient  and  more  consistent  with  the  job  if  the  student  had  been  required  to 
recall  the  steps  of  procedure  during  training.  If  training  time  is  limited,  the 
recall  performance  criterion  may  be  lowered  when  the  job  only  requires 
"general  familiarity,"  that  is,  when  students  are  permitted  to  perform  at  less 
than  100%. 

If  you  have  a  situation  in  which  recognition  is  appropriate  for  a  REMEMBER 
level  objective,  multiple-choice,  matching,  or  true-false  test  items  may  be 
appropriate,  even  if  the  content  type  is  concept,  procedure,  rule,  or  principle. 
These  do  not  appear  on  the  chart  above,  because  even  though  they  can  be 
used,  they  are  not  the  best  choice.  In  this  situation,  it  is  also  a  good  idea 
to  recheck  a  recognition  objective  to  make  sure  it  is  appropriate  for  the  job. 


Multiple-choice,  matching,  and  true-false  items  may  be  appropriate,  if 
carefully  designed,  for  many  USE-level  tasks.  For  example,  a  category 
classification  is  often  a  true-false  judgment.  If  the  student  must  solve  a 
math  problem  (Use-Rule),  a  multiple-choice  item  in  which  all  alternatives  are 
reasonable  is  appropriate.  Also,  some  USE-Principle  predictions  involve  a 
limited  set  of  possible  alternatives;  again,  multiple-choice  is  appropriate. 

The  reason  why  test  item  format  is  important  is  that  students  are  not  dumb! 
The  first  thing  most  new  students  do  in  a  course  is  find  out  how  they  will  be 
tested.  Then,  they  study  material  in  order  to  to  pass  the  tests.  If  the 
objective  requires  a  student  to  memorize  something,  multiple-choice  tests 
should  not  be  used,  because  students  will  learn  in  order  to  recognize,  but  not 
to  recall.  From  your  own  experience,  it  should  be  dear  that  students  study 
less  carefully  for  a  multiple-choice  or  true-false  test,  than  for  a  completion 
or  short-answer  test.  The  test  items  and  the  format  should  therefore  be  like 
the  tasks  the  student  will  do  on  the  job. 

5.1.2  A  Test  or  Test  Item  Missing  for  Some  Objectives  (TLO  or  LOs),  or  the 
Critical  Parts  of  the  LOs  (Questions  2  and  3). 

A  Terminal  Learning  Objective  (TLO)  is  a  specific  description  of  the  action 
with  conditions  and  standards,  to  be  performed  after  training.  A  Learning 
Objective  (LO)  is  an  objective  which  specifies  a  performance  which  is  a  part 


72 


of  or  a  prerequisite  to  the  action  specified  by  its  superordinate  learning 
objective  or  TLO.  Critical  parts/LOs  have  the  following  characteristics: 

•  They  are  some  of  the  most  measurable  and  observable  elements 
of  the  task. 

•  They  have  serious  consequences  of  inadequate  performance. 

•  They  are  common  sources  of  failure. 

•  They  are  actions  which  are  not  already  known  and  therefore 
must  be  taught. 

For  the  REMEMBER  task  level,  each  piece  of  information  is  critical.  Each 
of  the  following  should  be  tested: 

•  Facts 

•  Concept  characteristics 

•  Procedural  steps 

•  Steps  and  conditions  for  branching  in  a  rule 

•  Guidelines  in  a  principle. 

For  the  USE-AIDED  and  USE-UNAIDED  task  levels,  the  parts  of  the 
OBJECTIVE'S  ACTION  are  critical: 

•  For  CONCEPTS,  are  all  CRITICAL  CHARACTERISTICS  tested? 

Can  you  be  sure  the  student  knows  each  one  and  can  you  identify  which  ones 
the  student  does  not  know? 

•  For  PROCEDURES,  is  each  step  performed? 

•  For  RULES  and  PRINCIPLES,  is  each  step  performed?  Are  there 

items  for  all  possible  branches  or  decisions,  or  at  least  those 
that  would  commonly  occur  on  the  job? 


5.1.3  Test  Item  Task  Level  Imt 


Matched  With  Objective's  Task  Level 


The  task  level  of  the  test  item  (Remember,  Use-Unaided,  or  Use-Aided) 
should  match  the  task  level  of  the  objective.  This  means  that  the  action 
verb  in  the  test  item  should  be  the  same  as  the  action  verb  in  the  objective, 
or  at  least  the  same  behavior  must  be  required.  If  not,  the  test  item  is 
measuring  something  different  than  what  was  required  in  the  objective. 


5.1.4  Test  Item  Content  Imt 
(Question  5). 


Matched  with  Objective's  Content 


For  the  REMEMBER  task  level,  the  item  should  test  the  identical  information 
specified  by  the  objective.  For  the  USE-AIDED  and  USE-UNAIDED  task 
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levels,  the  action  of  the  items,  or  the  combined  actions  of  a  group  of  items, 
should  be  the  same  as  the  objectives  action. 

EXAMPLE:  Objective:  Given  the  necessary  tools  and  an  operator's  manual, 
the  student  will  set  up  and  operate  a  double-acting  reciprocating  pump,  in 
five  minutes  and  according  to  the  manual  specifications. 

Inconsistent  test  item:  "List  the  steps  of  procedure  for  starting,  operating, 
and  stopping  a  double-acting  reciprocating  pump." 

This  test  item  is  inconsistent,  because  its  task/content  is  REMEMBER 
PROCEDURE  instead  of  USE-AIDED-PROCEDURE.  Notice  that  the  action 
the  student  is  to  perform  in  the  test  is  not  the  same  as  the  action  required 
in  the  objective. 

Consistent  test  item:  "Use  the  operator's  manual,  and  necessary  tools  to  set 
up  and  operate  a  double-acting  reciprocating  pump.  You  will  pass  this  test  if 
you  complete  this  task  within  5  minutes,  in  accordance  with  the  manual 
specifications." 

5-1-5  Test  Item  Conditions  Improperly  Matched  with  the  Objective's 
Conditions  (Question  6). 

The  conditions  statement  in  the  training  objective  should  determine  the 
conditions  for  both  training  and  testing.  Consequently,  the  conditions  for  the 
training  and  testing  should  be  the  same.  If  the  problem  is  that  the  correct 
training  or  testing  conditions  cannot  be  set  up  for  some  reason,  find  out  why. 
If  material  and/or  facilities  required  by  the  lesson  plan  exist  but  are  not 
available  or  cannot  be  guaranteed,  suggest  that  training  and  testing  be 
redesigned  for  material  and/or  facilities  that  can  be  guaranteed.  Beware  of 
impossible  conditions,  i.e.,  "NBC"  conditions,  that  cannot  be  reproduced.  Such 
conditions  are  essentially  "no"  conditions  since  they  cannot  be  generated.  If 
the  test  conditions  are  more  difficult  than  the  training  conditions,  poor 
trainee  performance  is  probably  more  a  function  of  conditions  than  trainee 
learning  (since  the  trainees  have  not  had  a  chance  to  practice  under  these 
conditions).  If  the  test  conditions  are  easier  than  the  training  conditions, 
poor  trainee  performance  under  these  easier  conditions  almost  guarantees  poor 
trainee  performance  under  the  more  difficult  conditions  of  training.  In  either 
case  (test  conditions  easier  or  harder  than  training  conditions),  the  test  and/or 
training  conditions  should  be  brought  in  line  with  the  objectives.  If  the 
objectives  are  wrong,  they  should  be  changed. 

5-1*6  Test  Item  Standards  Improperly  Matched  with  the  Objective's  Standards 


First,  if  the  standard,  as  written  in  the  training  objective,  is  not  clear,  is 
missing,  or  does  not  specify  how  well  the  trainees  must  perform,  it  needs  to 
be  rewritten.  The  standard  must  specify  clearly  and  specifically  how  well  the 
trainees  must  perform  to  be  a  GO.  The  standard  must  have  real  meaning  for 
the  trainees.  It  must  be  communicated  to  them,  and  be  written  in  the  lesson 
materials,  in  words  that  they  understand  and  can  use  to  judge  their  own 
performance.  This  standard  should  be  used  in  the  final  stages  of  practice  (it 
is  the  standard  the  trainees  must  meet  in  practice)  and  in  the  test.  If  the 
training  and  testing  standards  are  not  the  same,  the  test  and/or  training 


standard  should  be  brought  in  line  with  the  objectives.  In  criterion-referenced 
testing,  standards  are  not  arbitrarily  selected.  It  makes  no  sense,  for 
example,  to  require  a  student  to  get  80%  of  the  items  right,  if  he  needs  to 
recall  all  the  information.  On  the  other  hand,  for  some  tasks,  a  70%  or  80% 
criterion  may  be  reasonable.  In  all  cases,  though,  the  standard  specified  in 
the  objective  should  be  used. 


5.2  TESTING:  Test  Instructions 


THE  PROBLEM.  The  instructions  given  to  the  trainees  at  the  beginning  of 
the  test  were  not  clear  enough  so  that  the  trainees  could  easily  understand 
what  they  were  expected  to  do  and  how  well  they  were  expected  to  do  it. 

This  may  have  occurred  because  the  test  administration  directions  were 
incomplete,  or  the  instructor  did  not  follow  them,  or  because  the  test 
instructions  provided  to  the  students  were  inadequate.  Test  instructions  must 
be  clearly  presented  to  the  trainees. 

EVIDENCE  for  ambiguous  test  instructions  will  come  from  the  following 
worksheets: 

Cl.U  Observation  of  a  Performance  Test 
C1.5  Observation  of  a  Written  Test 
C1.6  Observation  of  an  Oral  Test 

Question  21.  Are  test  administration  directions  complete? 

Question  22.  Do  instructors  follow  the  directions  when  administering  the 
test? 

Question  23.  Are  adequate  test  instructions  provided  to  the  student? 
RECOMMENDATIONS  for  training  program  modification. 

5.2.1  Incomplete  Test  Administration  Directions  (Question  21). 

Instructions  for  administering  the  test  were  incomplete  or  unclear  or  were 
missing  altogether.  The  instructions  for  administering  the  test  should  be 
written  so  as  to  insure  standardization  of  test  administration  procedures 
across  examiners.  The  lesson  plan  should  outline  precisely  how  the  test  is  to 
be  administered.  Included  should  be  the  placement  of  personnel  and 
equipment  and  test  administration  procedures  that  specify  the  testing  sequence 
and  guide  the  evaluator  in  testing  the  soldiers  on  each  of  the  tasks  to  be 
performed.  Such  guidance  for  the  evaluator  is  needed  in  order  to  insure  that 
each  soldier  is  tested  in  the  same  way. 

5.2.2  Instructors  Did  Not  Follow  Test  Administration  Directions  (Question  22). 


Test  administration  directions  exist,  but  the  instructor  did  not  follow  them. 
There  are  any  number  of  reasons  why  this  may  occur.  For  example,  if 
examiners  do  not  have  stopwatches,  rigid  time  standards  cannot  be  adhered 
to.  "Political"  pressures  may  force  instructors  to  allow  students  to  exceed 
time  standards  or  to  give  hints,  prompts,  or  cues  during  the  test  in  order  to 
obtain  the  required  percentage  of  "GCs".  Instructors  may  feel  that  they  know 
better  than  the  test  developers  how  the  test  should  be  administered.  If  lack 
of  compliance  is  due  to  resource  constraints,  these  should  be  noted,  and  the 
required  resources  obtained,  if  possible.  Problems  with  deliberate  departure 
from  established  test  procedures  and  standards  will  have  to  be  settled  with 
the  training  manager. 


5.2.3  Test  Instructions  Unclear  or  Inappropriate  lor  Item  Type  (Question  23). 


Test  instructions  get  the  trainee  in  the  right  mental  frame  of  reference.  To 
do  this,  they  have  to  be  crystal  clear  to  the  trainees.  They  should  also  get 
each  trainee  into  the  same  mental  frame  of  reference.  To  do  this,  they  have 
to  be  given  to  each  trainee  in  exactly  the  same  way.  This  is  why  test 
instructions  are  standardized  and  are  written  down  for  the  examiner  to  read. 
The  examiner  should  read  these  instructions,  just  as  they  are  written,  for 
each  trainee.  If  the  trainees  did  not  seem  to  understand  what  was  required 
of  them  after  hearing  the  test  instructions,  then  the  instructions  themselves 
may  need  to  be  rewritten.  Test  instructions  should  tell  the  trainee  what  he 
or  she  will  have  to  do,  the  conditions  they  will  have  to  perform  under,  the 
equipment/tools  they  will  have  and  can  use,  how  much  time  they  will  have, 
and  the  standards  for  successful  completion  of  the  task.  Any  rewritten 
instructions  should  be  "tried  out"  on  a  couple  of  trainees  who  are  typical  of 
normal  input  to  training. 

Guidelines  for  test  instructions  for  various  types  of  content  follow: 

Test  Directions  for  Memory  Level  Items  (i.e.t  Remember  -  Fact,  Concept, 
Procedure,  or  Rule) 

Each  test  item  (or  section)  should  tell  the  student  how  to  make  the  desired 
response  (e.g.,  "State  the  items  in  the  space  provided  below,"  "Circle  the 
appropriate  answer,"  "Label  the  controls  on  the  attached  illustration  by 
writing  the  name  next  to  each  relevant  control,"  etc.),  what  information  is 
given  in  the  item,  and  the  criteria  for  performance  (if  exact  order  is 
important,  exact  words  or  paraphrases  acceptable).  Directions  should  be 
complete  and  specific.  A  trainer  should  also  provide  adequate  space  for  a 
complete  response.  The  amount  of  space  needed  for  the  average  student  to 
make  a  complete  response  should  be  indicated  after  the  item.  For  fill-in-the- 
blank,  short-answer,  or  multiple-choice  items,  lines  should  be  drawn  to 
indicate  the  placement  and  approximate  length  of  answers. 

Test  Directions  for  Use-Concept  Items 

The  learner  should  know  exactly  what  to  do  with  the  information  in  each 
item.  Directions  should  specify  what  to  use  in  the  problem  (e.g.,  "Read  the 
name  of  the  concept  and  instances  listed  below  it."),  what  to  do  with  it  (e.g., 
'Identify  the  examples  of  the  concept."),  and  how  to  do  it  (e.g.,  "Place  the 
letter  of  the  answers  in  the  space  next  to  each  concept  name.").  In  addition, 
directions  should  tell  the  learner  of  any  criteria  pertaining  to  the  (e.g.,  "All 
examples  must  be  correctly  identified  for  credit  on  the  objective.").  It  is 
assumed  that  the  criterion  for  performance  is  100  percent  if  not  stated  in  the 
objective. 

Test  Directions  for  Use-Procedure  Items 

These  must  include  explicit  instructions  to  the  learner  telling  what  he  is  to 
do,  what  to  use  to  do  it,  what  limitations  there  are  on  performance,  and 
what  the  criteria  are  for  adequate  performance.  This  information  should  all 
be  contained  in  the  objective,  which  may  simply  be  repeated  or  rewritten  as 
a  specific  set  of  directions. 
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Test  Directions  for  Use-Rule  Items 


Directions  for  completing  the  item  should 

1.  Clearly  label  what  is  given  in  the  practice  situation. 

A  student  should  be  able  to  read  a  problem  and  know  what 
information  in  the  problem  is  needed  to  solve  it.  The 
information  which  is  given  to  the  student  should  have  been 
mentioned  in  the  conditions  of  the  objective.  In  addition,  when 
the  information  is  presented  in  the  problem,  it  should  be  clearly 
labeled  and  identified. 

For  example,  for  the  objective, 

Given  the  lengths  of  the  two  sides  of  a  right  triangle  adjacent 
to  the  right  angle,  and  the  appropriate  formula,  calculate  the 
length  of  the  hypotenuse  to  +  .01  accuracy  and  show  all  work, 

the  directions  to  the  test  item(s)  for  the  student  should  read, 

Using  the  formula  c  =  a^  +  b^  and  these  values  for  the  two 
sides: 

a  =  6  b  =  9 

find  the  hypotenuse  c,  showing  all  work  and  calculating  to  +  .01 
accuracy. 

> 

2.  State  the  outcome  to  be  produced  from  using  the  rule  and 
include  any  necessary  criteria  (as  stated  in  the  objective). 

The  directions  should  also  include  the  outcome  and  criteria  which 
were  stated  in  the  objective.  Using  the  example  stated  earlier, 
note  the  outcome  and  criteria. 

Using  the  formula  c  =  a^  +  b^  and  these  values  for  the  two 
sides: 

a  =  6  b  =  9 

find  the  hypotenuse  c,  showing  all  work  in  the  space  provided 
and  calculating  to  +  .01  accuracy. 

If  the  objective  requires  the  student  to  calculate  the  answer, 
then  the  item  should  not  require  a  different  behavior  (such  as 
selecting  the  correctly  worked  problem). 

3.  Specify  the  way  the  information  should  be  used. 


The  directions  specify  what  form  the  answer  should  take,  where 
to  place  it,  whether  or  not  to  show  work,  and,  if  required,  where 
to  show  it.  If  multiple-choice  items  are  used,  the  student  should 
be  told  whether  to  circle  his  answer,  write  the  response  (or 
responses)  in  the  given  blank,  etc. 


5.3  TESTING:  Test  Adequacy 


THE  PROBLEM:  Test  items  were  inadequately  designed.  Items  were  unclear, 
ambiguous,  not  well  constructed,  or  not  free  of  hints.  In  other  words,  the 
tests  may  have  been  invalid  and  unreliable.  If  a  test  does  not  accurately  and 
reliably  assess  trainee  performance,  you  have  no  way  of  knowing  whether  the 
trainees  can  actually  perform  (or  will  fail  to  perform)  when  placed  on  the 
job.  Put  another  way,  if  your  tests  are  bad,  your  test  scores  are 
meaningless. 
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EVIDENCE  for  inadequate  tests  will  come  from  the  following  worksheets: 

B2  Product  Evaluation  -  Evaluate  Test  Materials 

C1.4  Observation  of  a  Performance  Test 

C1.6  Observation  of  an  Oral  Test 

Question  8.  For  true-false,  multiple-choice,  and  matching  items  is  only 
one  answer  correct? 

Question  9.  For  short  answer,  fill-in,  listing,  and  performance  items 
are  all  acceptable  answers  in  the  answer  key? 

Question  11.  Is  the  language  of  test  item  easy  for  students  to 
understand? 

Question  12.  Is  the  test  item  different  from  previous  PRACTICE  and 


EXAMPLES? 


Question  13.  Is  the  answer  to  the  test  item  given  away  by  another  item 
or  group  of  items  on  the  test? 

Question  14.  Is  the  answer  to  the  test  item  dependent  on  answering  a 
previous  item  or  group  of  items  correctly? 

Question  15.  Are  sketches  and  diagrams  used  in  the  test  item  easy  to 
understand. 


Question  16.  Is  the  test  item  tricky  or  misleading  in  that  it  points  to 
an  incorrect  answer? 

Question  17.  Is  the  test  item  well-constructed? 

Question  18.  When  steps  are  scored,  does  the  instructor  use  a  checklist? 


Question  19.  On  selected  response  tests  (i.e.,  true-false,  multiple  choice, 
or  matching)  is  each  correct  answer  position  (e.g.,  a,  b,  c, 
or  d)  used  about  the  same  number  of  times? 

Question  20.  On  selected  response  tests  (i.e.,  true-false,  multiple  choice, 
or  matching)  are  specific  patterns  of  correct  answer 
positions  repeated  in  blocks? 


RECOMMENDATIONS  for  training  program  modification. 

5.3.1  More  Than  One  Answer  Correct  (Question  8). 

There  was  more  than  one  correct  answer  to  a  selected  response  (true-false, 
multiple  choice,  or  matching)  test  item.  The  item  is  therefore  likely  to  be 
difficult  for  the  student  to  answer  correctly. 

Here  is  an  inadequate  multiple-choice  item: 

If  you  are  driving  down  a  steep  grade,  you  would  be  likely  to  shift  gears 

a.  down  for  power. 

b.  up  for  power. 

c.  down  to  decelerate. 

d.  down  to  slow  the  car. 

The  item  is  ambiguous  because  choices  "c"  and  "d"  are  paraphrases  of  each 
other,  giving  two  possibly  correct  answers. 

Here  is  a  corrected  multiple-choice  item: 

If  you  are  driving  down  a  steep  grade,  you  would  be  likely  to  shift  gears 

a.  down  for  power. 

b.  up  for  speed. 

c.  down  to  decelerate. 

d.  up  to  reduce  engine  RPMs. 

5.3.2  Ail  Acceptable  Answers  Not  in  Answer  Key  (Question  9). 

There  were  one  or  more  correct  answers  to  a  short  answer,  fill-in,  listing  or 
performance  item  which  were  not  included  in  the  answer  key. 

Short  Answer 


The  answer  key  should  identify  allowable  alternatives.  If  only  one  exact 
answer  is  acceptable  for  a  short-answer  item,  it  might  be  more  appropriate  to 
convert  it  to  a  fill-in  item.  Short-answer  items  are  most  appropriate  when 
learners  are  to  have  some  flexibility  while  clearly  specifying  the  requirements 
for  a  correct  answer.  The  key  indicates  to  the  scorer  what  other  forms  of 
the  answer  are  acceptable,  e.g.,  "paraphrases  are  acceptable." 

Here  is  an  adequate  short  answer  item: 


Item:  Give  the  rules  for  computing  the  mean  of  a  set  of  scores  as  a 
formula.  No  credit  for  partially  correct  answers. 


sum  of  scores 
number  of  scores 


Key:  X  =  JL*  or^  the  mean  = 

"  V  t 

all  available  alternatives 


Listing 

Learners  should  not  be  penalized  for  having  different  wordings  of  the  correct 
answer  unless  exact  answers  are  necessary  and  called  for  in  the  directions. 
Any  acceptable  alternatives  should  always  be  included  in  the  scoring  key. 


Fill-in 


The  item  should  be  worded  so  that  only  one  word  or  phrase  correctly 
completes  the  sentence.  If  more  than  one  answer  may  be  correct,  it  might 
be  more  appropriate  to  convert  the  item  to  a  short-answer  format. 

Performance 


There  may  be  several  ways  of  deciding  whether  a  performance  is  adequate 
(e.g.,  all  steps  performed  completely  and  accurately  in  order  vs.  the  final 
outcome  of  the  procedure  achieved  vs.  varying  degrees  of  acceptable 
performance  for  different  steps).  Scoring  criteria  should  have  been  specified 
in  the  objective  which  should  be  included  in  a  performance  checklist.  If  the 
performance  objective  allows  for  alternative  behaviors  (e.g.  the  order  of  steps 
within  the  procedure  is  optional,  or  it  makes  no  difference  whether  all  of  the 
steps  are  performed  as  long  as  an  acceptable  final  product  is  produced),  then 
these  allowable  alternatives  should  be  specified  in  the  scoring  criteria. 


5.3.3  Test  Item  Contains  Too  Many  Difficult  Words  or  Long  Sentences 
(Question  11). 


The  test  item  contained  too  many  difficult  words  or  long  sentences  (other 
than  job-related  terminology)  for  the  reading  level  and  verbal  skills  of  the 
students.  The  purpose  of  a  test  item  is  to  assess  a  trainee's  skill  or 
knowledge,  not  his  reading  comprehension  or  his  vocabulary.  Therefore,  it  is 
important  NOT  to  exceed  the  student's  reading  level  (or  vocabulary  level  in 
an  oral  or  performance  test).  Test  items  should  use  words  that  the  student  is 
familiar  with,  and  should  contain  fairly  short,  uncomplicated  sentences. 
Guidelines  for  lowering  the  reading  dificulty  level  of  instruction  are  provided 
in  Section  3.6.3. 


5.3.4  Test  Item  Used  as  Previous  Practice  or  Example  (Question  12). 

A  test  item  for  a  Use- Aided  or  Unaided  Concept,  Rule,  or  Principle  was  the 
same  as  a  previously  used  example  or  practice  item.  All  test  items  (except 
those  testing  common  errors)  should  use  instances  not  previously  encountered 
by  the  learner  in  instruction.  For  example,  for  a  Use-Concept  test  item, 
when  the  learner  sees  an  example  or  non-example  for  the  second  time  it  is 
very  possible  that  he  will  remember  it,  rather  than  the  critical  characteristics 
to  classify  it.  For  this  reason,  previously  encountered  examples  and 
nonexamples  in  test  items  do  not  necessarily  indicate  that  the  concept  has 
been  learned,  no  matter  how  many  of  them  the  student  answers  correctly. 
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You  have  only  tested  his  ability  to  memorize  specific  examples  and 
nonexamples.  Recommend  to  the  developer  of  the  instruction  that  a  large 
enough  pool  of  items  be  developed  to  insure  that  different  items  can  be 
selected  for  use  as  examples,  practice,  and  test  items. 

5.3.5  Answer  Given  Away  By  Another  Item  or  Items  On  the  Test  (Question 

W. 

The  answer  to  a  test  item  was  disclosed  by  another  item  or  group  of  items  ! 

on  the  test.  Often,  test  items  are  developed  which  when  examined  , 

independently  are  found  to  be  adequate,  but  which,  when  placed  near  to  each 
other  on  a  test,  inadvertently  provide  hints  to  the  test-wise  student.  For  i 

example,  if  the  following  item  appears  on  a  test: 

I 

Ohm's  Law  is  E  =  IR.  If  I  =  200  ma.,  and  R  =  47K  ohms,  then  | 

E  = 


there  should  not  be  another  item  on  the  test  which  asks  the  student  to  recall 
Ohm's  Law. 

5.3.6  Answer  Depends  On  Answering  a  Previous  Item  or  Items  Correctly 
(Question  14)^ 

Previous  item(s)  must  be  answered  correctly  in  order  to  answer  the  test  item 
correctly.  Test  items  should  stand  alone.  Response  to  one  item  should  not 
be  contingent  upon  response  to  previous  items.  Consider  the  following  pair  of 
items: 

1.  Li§t  the  ten  insects  most  common  to  the  Southwest. 

2.  State  the  distinguishing  characteristics  of  each  insect  listed  in 
answer  to  Question  1. 

In  this  case  the  student  may  know  the  distinguishing  characteristics  of  all  ten 
insects,  but  may  do  poorly  on  Question  2  because  he  cannot  remember  the 
entire  list  called  for  in  Question  1.  In  effect  he  is  being  penalized  twice  for 
being  unable  to  perform  the  behavior  required  by  Question  1,  while  Question  2 
is  not  accurately  assessing  his  knowledge  of  the  distinguishing  characteristics. 


5.3.7  Unclear  Sketches  or  Diagrams  Used  in  the  Test  Item  (Question  15). 

Sketches  or  diagrams  used  in  the  test  item  were  difficult  to  understand  or 
confusing  to  the  students.  Such  difficulty  can  arise  when  the  graphic 
conflicts  with  something  the  trainee  is  used  to  or  has  seen  before,  when  it 
conflicts  with  the  directions  for  answering  the  test  item,  or  when  it  is  poorly 
designed.  It  is  not  the  purpose  of  this  paragraph  to  detail  how  to  design 
effective  graphics  for  use  in  test  items.  Problems  alluded  to  here  should  be 
described  to  training  development  and/or  graphics  personnel. 

5.3.8  Tricky  or  Misleading  Test  Item  (Question  16). 


The  test  item  was  tricky  or  misleading  in  that  it  pointed  to  an  incorrect 
answer.  Consider  the  following  example  of  a  "trick"  True-False  item: 


T  F  The  area  of  a  rectangle  4  ft.  by  3  ft.  is  equal  to  12  sq.  yds. 

This  apparently  true  statement  is  rendered  false  by  an  insignificant  detail. 
The  purpose  of  a  test  item  is  to  assess  student  skill  or  knowledge  of  the 
subject  matter,  not  his  ability  to  withstand  the  wiles  of  a  sneaky  test 
developer.  Trick  items  also  have  a  negative  effect  on  student  morale. 

Test  Item  Poorly  Constructed  (Question  17). 

The  test  item  was  not  well  constructed.  Different  criterion  apply  to 
different  item  formats.  The  test  item  violated  one  of  the  following  criteria: 


True-False 

a.  The  item  should  include  only  one  statement  (idea)  to  be  judged 
true  or  false.  If  an  item  contains  two  facts  or  pieces  of 
information,  one  of  which  is  true  and  the  other  false,  learners 
may  not  know  how  to  respond  to  the  item. 

For  example,  if  a  true-false  item  reads: 

t 

T  F  Carrot  seeds  should  be  planted  1/16"  deep  and  lettuce 
seeds  1/8"  deep. 

learners  may  be  confused  by  having  to  respond  to  both  facts.  Likewise,  if 
learners  answer  incorrectly,  you  do  not  know  which  piece(s)  of  information 
they  do  not  understand  correctly. 

b.  Negative  statements  and  double  negatives  should  not  be  used. 

Determining  the  truth  or  falsity  of  a  statement  that  contains  one 
or  more  negatives  is  confusing  at  best.  Consider  the  following 
item: 

T  F  It  is  not  appropriate  to  shift  from  a  higher  gear  to  a 
lower  gear  if  added  power  is  not  necessary. 

Learners  reading  this  item  must  attempt  to  translate  the  statements  into 
positive  form  before  answering  it.  Why  not  do  that  work  for  them?  You're 
not  trying  to  find  out  if  they  can  translate  complex  grammar;  you  want  to 
determine  whether  they  know  when  to  shift  gears. 

c.  The  item  should  be  short.  When  learners  must  read  a  very  long 
statement  before  determining  whether  it  is  true  or  false,  it  may 
be  difficult  for  them  to  pick  out  that  part  of  the  statement 
which  is  important.  Long  statements  are  also  more  likely  to 
suffer  from  the  other  problems  described  in  this  section. 

d.  The  item  should  not  be  so  obvious  that  it  can  be  answered 
correctly  on  the  basis  of  common  sense  alone.  For  learners  with 
any  knowledge  of  English  grammar,  the  following  item  would  be 
obviously  true. 
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T  F  An  adverb  is  a  word. 


By  asking  such  questions,  which  ail  learners  would  be  expected  to  answer 
correctly,  you  find  out  nothing  about  the  ability  of  learners. 

e.  Absolutes  like  "none11,  "never11,  "all",  and  "every"  should  not  be 
used.  These  key  words  provide  irrelevant  clues  which  allow  the 
student  to  guess  the  correct  answer  on  the  basis  of  test 
sophistication  when  he  really  does  not  know  the  answer.  Most 
items  which  contain  words  like  "always"  or  "never"  will  be  false 
since  there  are  usually  exceptions  to  every  rule. 

f.  The  item  should  not  contain  the  words  "some",  "and",  or 
"generally".  Since  there  are  exceptions  to  every  rule,  most 
items  which  include  words  like  "sometimes"  or  "possibly"  will  be 
true  since  these  items  will  be  the  exceptions. 

g.  The  item  should  deal  with  things  that  are  clearly  either  true  or 
false.  If  the  wording  is  ambiguous,  learners  may  be  answering 
questions  different  from  those  which  you  are  trying  to  ask. 

For  example,  a  true-false  item  which  reads: 


T  F  A  rhombus  is  a  geometric  figure  with  four  sides  of  equal 
length. 

would  be  true  if  it  was  interpreted  to  mean  that  a  rhombus  had  those 
characteristics  among  others.  A  rhombus  must  also  have  two  obtuse  angles 
and  must  be  in  one  plane.  The  item  would  be  false  if  it  was  interpreted  to 
mean  that  a  rhombus  was  defined  by  only  those  characteristics.  Note  that 
while  the  words  themselves  may  be  clear,  the  entire  statement  has  no  clear 
interpretation. 

Multiple  Choice 


a.  All  alternatives  should  have  the  same  grammatical  structure. 
The  following  item,  which  offers  alternatives  that  are  not 
grammatically  parallel,  is  an  example  of  how  one  should  not 
construct  a  multiple-choice  test  item: 

A  typical  malfunctoning  catalytic  converter  will  emit  a 

a.  It  gives  out  a  blue  gas 

b.  Whining  noise 

c.  Non-operatonal 

d.  Hard  to  say  without  more  information. 


84 


Such  items  often  confuse  the  student,  thus  increasing  the  chance  that  a 
student  who  has  learned  the  objective  will  still  respond  incorrectly.  In  the 
corrected  version  below,  the  alternatives  are  grammatically  parallel. 

A  typical  malfunctioning  catalytic  converter  will  emit  a 

a.  blue  gas. 

b.  whining  noise. 

c.  grating  sound 

d.  periodic  thud. 

b.  The  item  stem  should  be  worded  positively,  that  is,  words  like 


"no''  and  "not"  should  not  be  used.  Negatives  in  the  item  should 


be  avoided  if  at  all  possible.  If  negatives  must  be  used,  they 
should  be  highlighted.  If  negatives  within  the  stem  are  NOT 
highlighted,  or  in  some  way  emphasized,  learners  may  NOT 
notice  them  and  may  interpret  the  item  incorrectly. 

c.  Repetitive  phrases  should  be  placed  in  the  stem,  rather  than  in 


the  alternatives.  If  the  same  words  are  part  of  every 
alternative,  they  should  be  placed  into  the  stem  once.  The  item 


Carrot  seeds 


a.  should  be  planted  1/16"  deep. 

b.  should  be  planted  1/8"  deep. 

c.  should  be  planted  1/4"  deep. 

d.  should  be  planted  1/2"  deep. 

would  be  much  better  if  the  alternatives  were  only  the  numbers  and  the  rest 
of  the  information  was  included  in  the  stem. 

d.  Numerical  choices  should  be  listed  in  order  of  magnitude. 


On  the  Fahrenheit  thermometer  water  freezes  at: 

a.  10° 

b.  64° 


d.  0o 
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Here  is  the  corrected  version: 


a. 

0° 

b. 

10° 

c. 

32o 

d. 

64° 

e.  The  item  should  not  require  a  great  deal  of  reading.  Here  is  an 
example  from  dentistry  that  has  lengthy  alternatives: 

Healthy  gingiva  ("gums")  aid  in  the  self-cleaning  process  by  their 

a.  tendency  to  direct  food  particles  toward  occlusal  surfaces. 


b.  tendency  to  force  particles  away  from  the  proximal  space. 


c.  tendency  to  deflect  particles  away  from  free  gingiva  on 
the  interdental  papillae. 

d.  close  adherence  to  the  tooth  surface  below  the  height  of 
contour  of  the  cervical  enamel. 

The  above  question,  while  technically  correct,  takes  a  good  deal  of  time  to 
read  and  requires  some  intensive  study.  It  detracts  from  the  testirg  process 
where  a  great  many  items  can  be  asked  in  a  short  period  of  time. 

Therefore,  items  which  require  too  much  reading  should  be  avoided  whenever 
possible. 

f.  Wording  in  the  stem  should  be  clear  and  unambiguous,  so  that 

only  one  correct  answer  is  possible.  If  the  wording  of  the  stem 
allows  for  more  than  one  correct  answer,  whether  or  not  more 
than  one  of  those  answers  is  present  as  an  alternative,  the  item 
is  likely  to  be  difficult  for  the  learner  to  answer  correctly. 
Consider  this  example: 

This  figure  is  a 

a.  diamond. 

b.  parallelogram. 

c.  rhombus. 

d.  quadrilateral. 
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Any  of  the  given  answers  could  be  correct,  depending  upon  your 
interpretation.  In  order  to  limit  the  possibilities  for  the  correct  answer,  more 
information  is  required  in  the  stem. 

g.  The  article  "a"  or  "an11  should  not  be  at  the  end  of  the  stem.  If 
the  stem  and  one  or  more  of  the  alternatives  are  NOT 
grammatically  consistent,  the  number  of  actual  alternatives  is 
reduced.  For  example,  in  the  following  item: 

A  part  of  speech  which  modifies  a  verb  is  an 

a.  noun. 

b.  preposition. 

c.  adjective. 

d.  adverb. 

The  choices  "noun"  and  "preposition"  could  NOT  be  correct  alternatives  since 
"an  noun"  or  "an  preposition"  is  grammatically  incorrect.  A  better  item 
would  be  "which  of  the  following  is  a  part  of  speech  which  modifies  a  verb?" 


h.  All  alternatives  should  be  plausible  to  someone  who  doesn't  know 

the  answer.  If  an  alternative  is  NOT  plausible,  or  believable, 
nobody  is  likely  to  choose  it.  Common  misconceptions  make  the 
best  distractors  since  they  are  likely  to  be  chosen  by  learners 
who  do  NOT  know  the  correct  answer.  In  the  item  below, 
alternatives  "c"  and  "d"  are  not  plausible;  that  is,  few  people 
would  believe  they  are  correct.  Alternative  "b,"  on  the  other 

hand,  is  more  likely  to  be  selected  by  someone  who  didn't  know 

the  correct  answer. 

Which  of  the  following  is  the  chemical  symbol  for  gold? 

a.  Au 

b.  Go 

c.  $ 

d.  Gold 

i.  Alternatives  like  "A  and  B  only"  or  "All  of  the  above"  should  not 
be  used.  It  is  definitely  easier  to  put  one  of  these  in  as  a  final 
alternative  than  to  think  of  something  better,  but  they  are 
limited  in  their  usefulness  because  they  can  often  be  selected 
without  complete  knowledge  of  the  item  content.  "None  of  the 
above"  should  NOT  be  used  if  the  student  is  asked  to  select  the 
"best  answer"  rather  than  the  "correct  answer." 
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Alternatives  should  be  approximately  the  same  length. 
Alternatives  which  are  not  similar  in  length  can  cause  learners 
to  lean  toward  or  away  from  the  correct  answer.  A  longer 
answer  is  often  correct  because  it  has  additional  qualifications 
that  other  answers  omit.  For  example,  in  the  item  below, 
alternative  "e"  is  longer  than  the  others  and  of  a  different  form 
since  it  is  more  limited  and  thus  more  likely  to  be  correct. 

If  a  graph  is  symmetric  about  the  y-axis,  which  point(s)  lie(s)  on  the  graph? 

a.  (x,  y) 

b.  (x,  -y) 

d.  (-x,  -y) 

e.  (-x,  y)  for  every  (x,  y) 

k.  Paraphrases  or  synonyms  in  the  alternatives  should  be  avoided. 

A  paraphrase  or  synonym  for  the  correct  answer  would  also  be  a 
correct  answer,  so  paraphrases  should  be  avoided.  Opposites  are 
frequently  easier  to  come  up  with  than  other  alternatives,  but 
they  are  often  inappropriate,  unless  they  really  are  plausible 
alternatives. 

Matching 

a.  Directions  should  include  statements  of: 

1.  the  contents  of  each  column.  The  purpose  of  the  item  is 
clearer  to  the  learner  if  a  name  is  given  to  each  column. 


2.  the  basis  for  matching.  If  the  desired  basis  for  matching 
is  not  stated,  a  learner  could  conceivably  match  any 
relationship  which  might  exist  between  the  entries  in  the 
columns. 

3.  how  often  choices  may  be  used.  It  is  important  for  the 
learner  to  know  whether  each  choice  can  be  used  only 
once  or  more  than  once. 

4.  how  many  answers  are  possible  for  givens.  This  can 
sometimes  be  indicated  indirectly  in  the  wording  of  the 
directions  rather  than  stated  specifically. 

Below  is  an  adequate  set  of  directions  for  a  matching  item.  The  sections 
related  to  each  of  the  points  above  are  marked  with  the  corresponding  letter. 


Match  the  state  capitals  in  column  "A"  with  the  states 


in  which  they  are  capitals  in  column  "B".  There  will  be 
2 

only  one  answer  to  each  item  in  "A"  and  each  item  in  "B" 

4  3 

will  be  used  only  once. 

Leaving  out  any  one  of  these  sections  would  result  in  inadequate  directions. 

b.  All  choices  should  have  the  same  grammatical  structure.  (See 
Criterion  "a"  under  Multiple  Choice) 

c.  Numerical  choices  should  be  in  order  of  magnitude.  (See 
Criterion  "d"  under  Multiple  Choice) 

d.  The  item  should  not  require  a  great  deal  of  reading-  If  the 
wording  of  the  entries  is  long  or  difficult  to  interpret,  then  you 
are  testing  something  more  than  the  learner's  knowledge  of  the 
desired  relationship. 

e.  Choices  should  be  arranged  in  some  sensible  order,  for  example, 
alphabetically  or  logically.  The  following  is  a  well-written 
matching  item: 

Match  each  prefix  with  its  meaning.  Each  prefix  has  only  one 
meaning  and  each  meaning  corresponds  to  only  one  prefix. 


Meaning 

1.  amb 

a.  About 

2.  circum 

b.  Above 

3.  super 

c.  Across 

4.  trans 

d.  Against 

e.  Almost 

f.  Apart 

g.  Around 

f.  Unless  choices  can  be  used  more  than  once,  extra  entries  should 
be  included  in  the  choices  column. 

If  the  columns  are  of  equal  length  without  repeated  use  of 
entries,  learners  may  be  able  to  match  some  entries  by  the 
process  of  elimination.  If  the  entries  in  the  choices  column  can 
be  used  more  than  once,  the  givens  column  can  be  the  same 
length  or  longer  than  the  choices  column. 


g.  None  of  the  entries  in  either  column  should  appear  obviously 
different  from  others.  One  title  should  be  appropriate  for  all 
entries  in  a  column. 

If  you  cannot  put  a  brief  title  on  each  column,  then  you 
probably  have  more  than  one  class  of  things  represented  there. 
Different  classes  should  be  tested  separately.  For  example,  the 
list  below  includes  both  cities  and  states.  While  they  might  all 
be  classified  as  locations,  they  should  probably  not  appear  in  the 
same  column  of  a  matching  item. 

Locations 

J  Los  Angeles 
Cities  K  Houston 
^  Minneapolis 

Georgia 
New  Mexico 
Oregon 

h.  Each  alternative  should  be  a  plausible  answer  for  all  or  most  of 
the  problems.  (See  Criterion  "h"  under  Multiple  Choice.) 

Fill-in 

a.  The  blank  should  be  at  or  near  the  end  of  the  sentence.  When 
blanks  are  placed  near  the  end  of  sentences,  learners  have  a 
chance  to  get  most  of  the  meaning  of  the  items  before  they  see 
the  blanks.  This  helps  clarify  what  the  items  are  trying  to  ask. 


] 


States 


b.  One  and  only  one  word  or  phrase  should  correctly  complete  the 
item.  If  the  item  is  worded  so  that  more  than  one  answer 
makes  it  a  true,  complete  sentence,  any  such  answer  can  be 
justified  as  correct.  It  is  important,  therefore,  to  word  items  so 
learners  must  supply  the  intended  correct  answer  in  order  to 
complete  the  sentence  accurately.  For  example,  the  item 

Columbus  discovered  America  in  (1492). 

could  be  correctly  completed  by  any  of  a  number  of  words  or  phrases  such  as 
"a  ship,"  "the  Santa  Maria,"  "the  western  hemisphere,"  "looking  for  the 
Orient,"  or  "1492."  If  the  item  was  intended  to  measure  the  year  of 
discovery,  it  should  say  so.  A  better  item  would  be 

Columbus  discovered  America  in  the  year  (1492)  . 

c.  Multiple  blanks  should  be  avoided.  More  than  one  blank  per  item 
frequently  results  in  an  item  with  little  meaning.  Frequently, 
these  blanks  can  be  correctly  filled  in  a  number  of  ways.  For 
example,  in  the  item 


The  symbol  for  (silver) _ is  (Au). 

the  blanks  can  be  filled  in  correctly  with  any  pair  that  includes 
a  symbol  and  what  it  represents. 

d.  Blanks  should  require  key  words.  The  word  left  out  of  the 
sentence  determines  what  the  item  is  measuring.  If  trivial  words 
are  omitted,  trivial  things  are  being  measured.  It  is  also 
important  that  the  correct  answer  NOT  be  a  "magic"  word  which 
could  be  supplied  by  a  learner  without  real  understanding  of  the 
question.  For  example,  in  the  item 

Larger  plants  will  usually  be  planted  farther  apart  than  (smaller) 
plants. 

the  word  "smaller"  is  probably  the  first  answer  most  people 
would  think  of  whether  or  not  they  know  anything  about  plant 
spacing.  The  key  words  in  this  item  are  probably  "farther 
apart."  Key  words  will  be  determined  by  what  the  item  is 
intended  to  measure. 

e.  There  should  be  no  grammatical  cues  to  the  correct  answers. 
Grammatical  cues  can  sometimes  help  a  learner  who  does  NOT 
know  the  correct  answer  eliminate  a  set  of  possible  wrong 
answers.  In  the  item 

A  sentence  must  contain  at  least  (one)  noun  or  noun 
substitute. 

a  learner  who  thought  a  sentence  had  to  have  two  nouns  would 
know  what  was  wrong  since  "noun"  in  the  sentence  is  singular. 

Short  Answer 


a.  The  required  answer  should  be  short.  Short  answers  consist  of 
either  phrases  or  one  or  two  short  sentences  which  supply  the 
necessary  information. 

b.  Directions  to  the  student  should  specify  how  the  items  will  be 
scored  without  giving  cues  to  the  correct  answers.  This  provides 
the  learner  with  exact  information  concerning  how  performance 
will  be  graded.  It  includes  such  things  as  number  of  points  per 
portion  of  the  answer,  whether  partial  credit  is  given  for  partial 
answers,  or  whether  the  entire  answer  must  be  given  for  any 
credit. 

c.  The  scoring  key  should  identify  allowable  synonyms  or 
alternatives.  If  only  one  exact  answer  is  acceptable  for  a  short- 
answer  item,  it  might  be  more  appropriate  to  convert  it  to  a  fill- 
in-the-blank  answer  item.  Short-answer  items  are  most 
appropriate  when  learners  are  to  have  some  flexibility  while 
clearly  specifying  the  requirements  for  a  correct  answer.  The 
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key  indicates  to  the  scorer  what  other  forms  of  the  answer  are 
acceptable. 


The  question  should  be  complete  with  enough  information  for  the 
learner  to  generate  an  appropriate  answer.  The  question  should 
provide  enough  information  for  the  learner  to  know  what  is 
expected  of  him.  It  should  tell  him  clearly  what  information  he 
should  include  in  his  short  answer. 

Credit  should  be  given  for  each  correct  part  of  an  answer.  This 
is  best  accomplished  by  having  a  checklist  of  all  the  points  in  a 
good  answer.  For  every  correct  portion  of  an  answer,  the 
student  earns  a  number  of  points.  For  example,  in  the  question 
"State  four  rules  for  safe  encampment  in  a  forested  terrain,"  we 
might  assign  one  point  for  each  rule  given  that  is  correct. 

Questions  should  be  clear  so  that  all  learners  will  have  the  same 
interpretation.  All  students  should  generally  interpret  what  the 
question  is  asking  in  the  same  way.  Being  specific  is  one  way 
to  achieve  this. 

BAD:  Discuss  safe  encampment. 

BETTER:  State  four  rules  for  safe  encampment  in  a 

forested  terrain. 

A  model  answer  should  be  written  for  each  question  as  an  aid  to 
scoring,  or  a  checklist  of  attributes  of  a  good  answer  should  be 
provided  for  use  in  scoring. 

Writing  a  model  answer  often  helps  the  test  item  developer  to 
understand  the  strengths  and  weaknesses  of  the  question  and  can 
be  the  first  step  toward  creating  a  checklist  of  attributes  of  a 
good  answer.  This  checklist  will  help  in  scoring  student 
responses.  It  is  desirable  to  give  students  these  model  answers 
following  the  test  so  that  they  may  better  evaluate  their 
performance. 

Items  should  be  assigned  points  according  to  their  importance. 

Point  values  should  be  made  known  before  the  test.  Let  the 
students  know  how  much  each  item  will  count  before  the  test. 

Items  should  be  scored  anonymously  (scorer  should  grade  students' 
papers  without  knowledge  of  their  identity.  Knowing  who  wrote 
the  response  often  biases  the  scoring. 

Content  should  be  scored,  not  grammar,  punctuation,  spelling  or 
penmanship  (unless  specified  in  the  objective).  Students  may  use 
poor  grammar,  punctuation,  or  spelling  or  have  poor  penmanship 
in  writing  their  responses.  These  factors  should  not  bias  scoring 
of  the  response.  The  scorer  should  be  reminded  that  it  is 
knowledge  that  is  being  tested  -  not  these  other  factors. 
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Listing 


a.  The  directions  should  specify  the  number  of  things  to  be  listed 
(if  appropriate  for  the  objective,  and  if  the  number  of  things  is 
not  a  clue£  If  scoring  will  be  on  the  bases  of  a  specific  length 
list,  learners  will  be  more  likely  to  correctly  answer  the  question 
if  they  know  the  length  of  the  list  expected. 

b.  The  directions  should  indicate  how  the  item  will  be  scored 
without  giving  cues  to  the  correct  answer.  Students  should  know 
if  they  must  list  items  in  a  certain  order,  if  they  must  use  the 
exact  words  or  presentation  given  in  the  generality,  and/or  if 
some  parts  of  the  list  will  receive  more  weight  when  scored. 

They  should  not,  however,  be  given  any  information  which  will 
allow  them  to  generate  part  or  all  of  the  list  from  reading  the 
directions  alone. 

c.  If  order  is  important,  the  scoring  key  should  treat  sequence 
separately.  If  sequence  is  not  judged  separately  from 
completeness,  then  one  thing  out  of  order  could  result  in  no 
credit  for  an  answer  which  is  complete,  but  out  of  sequence. 

d.  The  scoring  key  should  identify  allowable  synonyms  or 
alternatives.  Learners  should  not  be  penalized  for  having 
different  wordings  of  the  correct  answer  unless  exact  answers 
are  necessary  and  called  for  in  the  directions.  Any  acceptable 
alternatives  should  always  be  included  in  the  scoring  key. 

e.  Spelling  should  not  be  scored  unless  required  be  the  objective. 
(See  Criterion  "k."  under  Short  Answer.) 

f.  The  scoring  key  should  identify  relative  weights  of  different 
items  on  the  list,  if  appropriate.  Most  listing  items  will  give 
equal  weight  to  each  item  on  the  list.  If  some  parts  of  the  list 
will  recieve  more  weight  than  others,  this  should  be  made  clear 
in  the  scoring  key  and  also  in  the  directions. 

Performance 

a.  The  test  should  score  all  aspects  of  task  performance  and 

qualities  of  the  resulting  task  product  that  are  specified  in  the 
objective  and  no  others.  A  performance  item  will  assess  either 
task  performance  or  the  qualities  of  the  resulting  task  product. 

A  product  is  a  tangible  result  that  you  can  touch,  see,  smell, 
and/or  hear  following  an  activity  by  students.  A  performance  is 
a  sequence  or  set  of  actions  that  students  do  that  must  be 
observed  in  action  because  the  behavior  itself  is  the  final  result 
of  interest. 


The  following  is  an  objective  which  requires  evaluation  of  student 
performance: 


Given  an  emergency  procedure  simulator  and  a  cue  from  the 
instructor  that  there  is  a  partial  failure  of  your  parachute  (Model 
222),  correctly  execute  the  emergency  procedure  for  overcoming 
such  a  failure  within  five  seconds. 

Here  is  an  objective  requiring  evaluation  of  a  product: 

Given  an  NBG  3  (Nuclear)  report,  1:50,000  FORCHHEIM  Map 
Sheet,  overlay  paper,  grease  pencils,  a  coordinate  scale,  and  FM 
3-22,  plot  a  detailed  fallout  prediction  with: 

1.  All  coordinates  and  marginal  information  corresponding  to  the 
information  given  in  the  report. 

2.  The  radial  lines  drawn  to  within  +  2  degrees. 

3.  The  time  of  arrival  lines,  Zone  I  and  Zone  II  downwind  distances, 
and  cloud  radius  corresponding  to  line  ZULU  of  the  report. 

When  objectives  appear  that  cannot  be  classified,  the  objective  probably  needs 
improvement  and  lacks  sufficient  conditions,  standards,  or  behavioral 
terminology  to  help  you  identify  what  type  of  measurement  is  needed.  If  the 
statement  of  the  objectives  is  not  clear,  go  to  a  subject  matter  expert  or  the 
author  of  the  statement  and  ask  "Are  you  interested  in  how  the  student  does 
this  (performance),  or  in  a  result  (product)  that  is  produced?"  Thus,  in  many 
instances  you  cahnot  identify  the  appropriate  test  format  until  the  statement 
of  the  objective  is  improved. 

b.  The  directions  should  clearly  explain  what  the  student  is  to  do 
and  how  the  item  will  be  scored  (and/or  the  test  administrator 
should  clearly  communicate  that  to  the  student.  An  adequate 
performance  item: 

1.  Provides  sufficient  information. 

If  the  wording  is  such  that  the  learner  cannot  figure  out 
what  he  or  she  is  supposed  to  do,  the  learner  will  be 
unable  to  perform  the  task. 

2.  Matches  the  objective. 

Well-written  behavioral  objectives  usually  specify  the 
behavior  or  task  the  student  is  to  perform,  the  conditions 
under  which  the  student  is  to  perform  that  task,  and  the 
criterion  for  acceptable  performance.  A  well-written 
performance  test  item  should  match  each  of  these 
components  as  they  are  stated  in  the  objective.  For 
example,  if  the  objective  states  that 


The  student  will  add  with  100  percent 
accuracy,  and  without  the  aid  of  a 
calculator,  two  columns  of  four,  two- 
digit  numbers. 

then  the  test  item  might  look  like  this: 

Add  both  these  columns  of  numbers. 
(Note:  you  may  not  use  a  calculator.) 


32 

87 

86 

55 

43 

68 

94 

72 

Even  if  the  behavior  or  task  were  the  same  as  that  stated 
in  the  objective,  the  test  item  would  not  be  considered 
adequate  if  the  statement  concerning  nonusage  of  the 
calculator  was  not  present. 

States  the  required  behavior  unambiguously. 

The  behavior  stated  in  an  objective  should  reflect  that 
behavior  or  task  expected  of  the  learner  after  completing 
a  segment  of  instruction.  For  example,  identify,  classify, 
add,  and  subtract  are  such  behaviors.  Unfortunately,  these 
behaviors  are  also  impossible  to  observe.  They  can  all  be 
done  without  any  accompanying  overt  act  signaling  that 
they  have  taken  place.  We  have  all  experienced 
identifying  a  particular  object  or  phenomenon  without 
exclaiming  that  we  have  done  so.  Though  it  is  acceptable 
in  everyday  living  not  to  be  provided  with  constant 
indications  of  our  mental  prowess,  it  is  not  acceptable  in 
the  case  of  performance  measurement.  Therefore,  along 
with  each  of  these  covert  behaviors  must  appear  another 
behavior  which  describes  how  each  covert  behavior  will 
represent  itself.  For  example,  an  objective  may  state 

Without  the  aid  of  any  resource 
material,  identify  cats  with  100  percent 
accuracy  by  pointing  to  all  those 
appearing  in  a  field  of  fifty  animals. 

As  a  consequence  of  instruction  related  to  this  objective, 
we  would  like  for  the  student  to  be  able  to  identify  cats. 
The  student  will  indicate  to  us  whether  or  not  he  or  she 
can  identify  cats  by  pointing  to  them. 
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In  writing  performance  test  items,  if  the  primary  behavior 
(e.g.,  identify,  classify,  add,  etc.)  in  the  objective  is 
specified  alone  and  is  not  of  an  observable  nature,  the 
test-item  writer  can  specify  any  means  by  which  the 
student  must  show  ability  to  perform  the  priamry  behavior. 
If  both  a  primary  and  a  secondary  behavior  (e.g.,  in 
writing,  orally,  pointing,  etc.)  are  specified,  then  the  item 
in  the  objective  must  be  employed  in  writing  the  test 
item.  If  the  objective  only  specifies  a  secondary  behavior, 
then  the  test  developer  should  make  every  effort  to  secure 
information  concerning  the  primary  behavior  in  order  to 
insure  that  the  item  to  be  written  will  clearly  indicate  the 
student's  ability  to  perform  that  primary  behavior. 

Finally,  both  primary  and  secondary  behaviors  should  be 
clearly  specified  in  the  test  item. 

4.  Provides  the  conditions. 

The  conditions  under  which  the  behaviors  will  occur  must 
be  clearly  provided  and  accompany  each  test  item. 
Conditions  may  take  the  form  of  problems  to  be  solved  by 
the  learner.  They  should  be  provided  along  with  the  stem 
of  the  test  item.  In  some  cases,  however,  conditions  will 
indicate  under  what  circumstances,  or  provided  with  what 
additional  tool,  the  learner  is  expected  to  perform.  This 
statement  of  conditions  is  for  the  benefit  of  the  test 
administrator  who  must  insure  that  these  conditions  (e.g., 
with  the  aid  of  a  calculator,  given  a  tennis  racket,  etc.) 
stated  in  the  objective  (and  test  item)  exist  at  the  time 
of  testing. 

5.  States  the  criterion. 

The  criterion  must  be  clearly  stated  for  the  benefit  of  the 
student  as  well  as  the  test  grader  or  interpreter.  The 
student  should  be  informed  of  how  well  he  or  she  will 
need  to  perform  a  given  task  before  being  asked  to 
perform  it.  This  diminishes  the  possibility  of  any 
misunderstanding  on  the  part  of  the  student  or  the 
instructor  as  to  how  well  the  student  needs  to  perform  in 
order  to  be  considered  mastering  the  task. 

Each  step  (in  a  checklist)  should  be  self-explantory  to  anyone 
who  would  perform  the  task.  You  might  give  the  checklist  to  a 
beginning  trainee  and  ask  him  to  go  through  the  performance 
step  by  step.  If  the  checklist  is  well-constructed  he  should 
understand  the  steps  and  directions.  Some  items  in  the  checklist 
will  not  be  able  to  be  performed  without  some  teaching.  You 
might  have  available  a  manual  that  describes  in  detail  how  to 
carry  out  a  particular  item. 


I 

I 

! 


I 

You  could  also  give  the  checklist  to  a  third  party  and  ask  him 

to  judge  the  performance  of  a  skilled  expert.  Ask  that  person 

to  judge  (1)  when  a  step  is  completed  or  not,  and  (2)  whether 

each  step  is  completed  correctly.  Emphasize  that  the  checklist, 

not  the  performer,  is  on  trial.  This  process  should  help  to  < 

identify  those  words  or  phrases  that  are  unclear  in  the  checklist.  j 

d.  Each  step  (on  a  checklist)  should  be  stated  as  a  single  action;  j 

important  or  complex  decisions  should  be  presented  as  single 

steps.  Stating  the  steps  as  single  actions  makes  it  easier  to 
insure  that  they  are  performed  correctly  and  that  portions  of  a 
step  are  not  overlooked.  Sometimes,  however,  a  series  of  steps 
which  the  learner  already  knows  or  which  are  taught  elsewhere 
can  be  grouped  and  referred  to  in  a  single  step.  If  not,  the 
steps  must  be  stated  as  substeps  to  the  main  step  on  the 
checklist.  In  addition,  if  steps  are  performed  simultaneously, 
they  may  be  combined  into  a  single  step. 

In  listing  the  steps  on  a  checklist,  it  is  easy  to  overlook  those 
steps  which  require  some  mental  activity  and  include  only  those 
actions  which  are  easily  observed.  However,  it  is  important  to 
the  learner  that  any  important  or  complex  decisions  be  listed  as 
separate  and  distinct  steps.  In  some  instances,  the  decision  step 
leads  to  alternate  paths  in  a  procedure.  For  example,  a  decision 
step  might  be  "Determine  if  the  patient  is  in  shock."  Later,  the 
checklist  might  state  "If  in  shock,  keep  patient  quiet;  if  not, 
have  patient  move  to  cover." 

e.  The  scoring  key  should  specify  all  criteria  the  performance  must 
meet,  such  as  completeness,  accuracy,  quality,  time  limit,  rate, 
etc.  The  scoring  key  should  specify  the  same  criteria  for 
minimally  acceptable  task  performance  that  were  contained  in 
the  training  objective.  These  standards  may  consist  of: 

1.  Time/speed,  if  it  is  essential  that  a  task  be  performed  at 
or  within  a  specified  time,  for  a  particular  duration,  or  at 
a  specified  rate  of  speed  (e.g.,  within  5  seconds). 

2.  Accuracy /error  rate,  if  precise  numerical  measurements 
can  be  made  of  the  accuracy  of  task  performance  (other 
than  tolerance  limits)  or  of  the  permissable  number  of 
errors  (e.g.,  8  out  of  10  correct). 

3.  Adherence  to  authority  or  established  procedure,  if  a 
publication,  manual,  or  regulation  explicitly  describes  the 
proper  procedure  the  student  must  execute,  and  that  is  to 
be  the  performance  standard  (e.g.,  in  accordance  with  FM 
21-41). 

4.  Performance  Precision.  When  a  performance  can  be 
executed  within  an  acceptable  tolerance  range  or  limit 
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which  can  be  readily  measured,  the  range  of  variation  j 

should  be  noted  (e.g.,  within  ♦  5°).  | 

5.  Conformity  with  quality  specification.  When  performance  | 

cannot  be  precisely  measured,  expert  judgment  is  often  i 

used  to  assess  the  quality  of  the  performance  or  final  ' 

product.  When  qualitative  criteria  can  be  specified, 

they  should  be  noted  (e.g.,  a  correct  description  will 
include  .  .  .). 

6.  Computational  accuracy.  If  the  task  is  of  a  computational  j 

nature,  acceptable  limits  should  be  noted  within  which  the 

answer  will  be  accepted  as  correct  (e.g.,  gross  weight 
calculated  within  50  pounds). 

7.  Safety  considerations,  if  a  specific  safety  consideration 
must  be  complied  with  (e.g.,  without  radiation  hazard  to 
personnel). 

8.  Completion  cues,  if  the  criterion  for  correct  task 
performance  is  the  creation  of  a  specific  condition  or 
product  (e.g.,  to  arrive  at  the  landing  platform  on  a  10 
foot  hover). 

f.  If  steps  in  the  performance  are  scored,  a  checklist  should  be 
provided.  Checklists  should  be  the  method  of  choice  for 
performance  evaluation  when  a  set  or  sequence  of  observable 
behaviors  in  a  performance  or  product  must  be  examined. 

A  thecklist  is  literally  a  list  of  behaviors  that  can  be  checked 
by  an  observer.  It  can  be  a  simple  list  of  a  few  sequential  and 
related  elements  or  a  complex  observational  system  involving 
precise  behavioral  definitions  for  each  element.  In  either  case, 
it  is  crucial  that  checklists  be  developed  from  a  job  analysis  and 
field  tested  with  actual  students. 

g.  Any  rating  scales  should  be  specific  enough  so  that  all  raters 
will  interpret  them  in  much  the  same  manner.  Rating  scales 
often  involve  abstract  qualities.  In  these  cases,  the  scale  is 
created  and  labeled  with  words  such  as  originality,  durability, 
etc.  These  terms  may  vary  in  meaning  between  raters.  The 
problem  again  is  one  of  definition.  Any  time  we  deal  with  an 
abstraction,  it  is  the  responsibility  of  the  instructor  or  designer 
of  instruction  to  clarify  the  standards  and  objectives.  The 
following  is  an  example  to  illustrate  the  distinction  between 
clearly  stated  traits  and  those  that  are  less  clear. 

Bad  Example;  Rate  the  originality 
expressed  in  the  following  photographs. 


Better:  Rate  the  originality  in  the 
following  photographs.  Originality  is 
defined  as  (1)  use  of  a  wide  range  of 
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tones,  (2)  nonsymmetrical  composition, 
and  (3)  realism.  Each  will  be  explained 
in  detail. 


h.  All  raters  should  agree  on  the  quality  of  task  performance. 

Lack  of  agreement  among  raters  is  a  problem  most  Americans 
have  observed  in  viewing  the  Olympic  Games  in  gymnastic  and 
diving  competition.  Different  experts  may  often  disagree  about 
the  quality  of  performances,  and  sometimes  there  may  be  lack  of 
agreement  for  a  single  rater  viewing  the  same  people  on  several 
different  occasions.  One  of  the  most  effective  ways  to 
overcome  this  problem  is  to  clearly  define  the  objectives  of  the 
performance.  If  the  qualities  of  the  performance  to  be  rated 
are  well  defined,  raters  will  converge  in  their  judgments  and 
ratings  will  be  reliable.  Another  preventive  action  is  to  train 
raters  before  they  start  rating.  If  there  is  lack  of  agreement, 
the  training  session  should  focus  on  problem  raters  or  the  process 
of  rating  as  a  whole. 

5.3.9  Checklists  Not  Used  (Question  18). 


Checklists  were  appropriate  for  scoring  task  performance,  but  were  not  used 
by  the  examiners.  It  is  assumed  that  the  checklist  meets  the  criteria  for 
adequacy  is  defined  in  the  Performance  section  of  the  Recommendations  for 
Question  17.  If  it  does  not,  then  the  checklist  needs  to  be  redesigned.  The 
examiner  should  use  the  checklist.  If  he  or  she  feels  that  the  checklist  is 
inaccurate  or  unusable,  then  he  or  she  should  take  steps  to  have  it  modified. 
The  solution  to^an  inadequate  checklist  is  not  just  to  stop  using  it.  If  the 
examiner  has  difficulty  following  the  checklist  because  he  or  she  has  not  be 
trained  in  its  use,  a  training  session  should  be  immediately  scheduled. 

5.3.10  Some  Correct  Answer  Positions  Used  Too  Frequently  (Question  19). 

On  a  selected  response  test  (true-false,  multiple  choice,  or  matching)  some 
correct  answer  positions  (e.g.,  a,  b,  c,  or  d)  were  used  more  than  others.  If, 
for  example,  it  becomes  obvious  to  the  test  takers  that  the  instructor  has  a 
tendency  to  make  "d"  the  correct  answer  to  multiple  choice  questions,  they 
will  select  "d"  when  the  do  not  know  the  anwer.  This  could  artificially 
inflate  their  scores.  If  this  problem  is  discovered,  recommend  to  the 
instructor  that  correct  answer  positions  be  used  about  equally  often. 

5.3.11  Correct  Answer  Positions  Form  Repetitive  Pattern  (Question  20). 

On  selected  response  tests  (true-false,  multiple  choice,  or  matching)  specific 
patterns  of  correct  answer  positions  are  repeated  across  test  items  (e.g.,  c,  d, 
a,  b,  c,  d,  a,  b,  etc.)  or  single  positions  are  repeated  in  blocks  (e.g.,  c,  c,  c, 
c,  d,  d,  d,  d,  etc.).  This  can  give  students  clues  that  make  the  test  easier 
than  it  should  be.  If  this  problem  is  discovered,  recommend  to  the  test 
developer  that  the  answer  positions  be  randomly  assigned  so  that  no  pattern 
can  be  discerned. 
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5.4  TESTING:  Test  Realism 


THE  PROBLEM.  The  test  did  not  accurately  reflect,  as  closely  as  possible, 
the  requirements  and  conditions  of  the  job.  The  test,  of  course,  is  directly 
related  to  the  training  objectives.  The  training  objectives,  however,  should 
specify  the  tasks  and  conditions  in  such  a  manner  that  the  job  requirements 
are  reproduced  as  accurately  as  possible  within  the  training  environment. 

Such  training  objectives  will  guarantee  job  related  testing  so  long  as  the  test 
follows  the  objectives  closely. 

EVIDENCE  for  a  lack  of  test  realism  will  come  from  Worksheets  Cl. 4 
Observation  of  a  Performance  Test,  and  Cl. 5  Observation  of  a  Written  Test. 


Question  10.  Does  the  test  item  provide  opportunities  for  COMMON 
ERRORS  to  be  made? 

Question  24.  Does  the  FINAL  TEST  integrate  tasks  as  they  are 
integrated  in  the  "real  world?*' 

Question  25-  Are  tasks  and  task  steps  tested  in  the  same  sequence  as 
they  are  performed  in  the  "real  world?" 

RECOMMENDATIONS  for  training  program  modification. 

5.4.1 


Possible  responses  to  the  test  item  did  not  allow  students  to  choose  or 
commit  most  errors  common  in  the  real  job  task.  Test  items  should  be 
written  so  as  to  permit  students  to  make  common  errors.  For  example,  the 
wrong  answers  (called  detractors  or  foils)  on  multiple-choice  tests  should 
reflect  typical  student  misconceptions.  Performance  tests  should  not  be 
constrained  so  as  to  preclude  students  making  common  mistakes  (see  also 
TESTING:  Contamination). 

Identification  of  common  errors  often  requires  consultation  with  a  subject 
matter  expert.  His  familiarity  with  the  task  or  objective  should  enable  him 
to  think  of  the  common  mistakes  which  trainees  make  when  performing  the 
task  or  the  behavior  required  by  the  objective.  If,  for  example,  the  objective 
involves  applying  a  rule,  common  mistakes  might  involve  things  like: 

1.  Confusing  one  type  of  data  with  another  (e.g.,  mixing  similar 
symbols,  "C”  or  "Cl"  in  chemical  formulas). 

2.  Confusing  the  operations  involved  in  using  the  rule  (e.g.,  adding 
instead  of  subtracting). 

3.  Misreading  tables  or  other  presentations  of  the  data  (e.g.,  taking 
information  from  the  wrong  column  or  row  in  a  table,  etc.). 

4.  Being  unable  to  use  difficult  forms  of  data  (e.g.,  numbers  in 
fraction,  large  decimal,  or  square  root  form). 


pport unities  Not  Provided  for  Common  Errors  To  Be  Made  On  Test 
Question  10). 
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5. 


Making  computational  errors  (e.g.,  misplacing  decimals  or  making 
common  multiplication  errors). 


5.4.2  Test  Requirements  Differ  from  Job  Requirements  (Question  24). 


The  final  test  did  not  integrate  tasks  as  they  are  integrated  in  the  real 
world.  The  test  should,  as  closely  as  possible,  measure  performance  as  it 
would  occur  under  actual  job  conditions  and  should  require  trainees  to  perform 
tasks  together  that  would  be  performed  together  on  the  job.  This  says  that 
the  test  should  approximate  job  requirements.  The  extent  to  which  this  can 
be  done  may  be  limited  by  resource  constraints,  but  such  constraints  can 
often  be  overcome  with  a  little  ingenuity.  Talk  with  the  training  observers 
to  find  out  why  they  rated  Question  24  a  "2"  or  "3."  Take  their  observations 
to  the  training  developer  to  see  if  a  more  realistic  test  can  be  developed. 


5.4.3  Tasks  or  Task  Steps  Tested  Out  of  Sequence  (Question  25). 


The  test  did  not  require  trainees  to  perform  tasks  or  task  steps  on  the  same 
sequence  as  they  are  performed  in  the  real  world.  Steps  for  a  procedure,  for 
example,  should  be  tested  in  the  same  order  in  which  they  were  presented 
and  practiced.  This  should  be  the  exact  order  in  which  they  are  best 
performed.  On  rare  occasions  the  order  of  certain  steps  within  a  procedure 
may  be  optional,  and  should  be  indicated  as  such  on  the  check  list. 


5.5  TESTING:  Contamination 


i 

THE  PROBLEM.  Trainees  received  help  during  the  test  in  the  form  of  hints,  j 

prompts,  or  cues  from  the  examiner.  The  test  should  be  a  measure  of  how  i 

well  trainees  can  perform  tasks  without  any  help  from  outside  sources.  The  j 

trainees  should  be  on  their  own  during  the  test  and  each  trainee  should  I 

receive  exactly  the  same  test. 

EVIDENCE  for  contamination  in  test  scores  will  come  from  the  following 
work  sheets: 

Cl. 4  Observation  of  a  Performance  Test 
C1.5  Observation  of  a  Written  Test 
Cl. 6  Observation  of  an  Oral  Test 

Question  26.  Is  the  test  free  of  external  cues  and  help? 

RECOMMENDATIONS  for  training  program  modification. 

5.5.1  Trainees  Got  Help  From  Examiners  (Question  26). 

That  trainees  should  not  get  help  from  the  examiners  seems  obvious,  but  it 
happens.  Examiners  are  not  always  committed  to  rigorous  measurement  of 
trainee  performance.  If  rigorous  measurement  is  not  required  by  training 
management  personnel,  it  will  not  be  exercised  by  test  examiners.  If 
examiners  are  also  the  trainers,  they  will  probably  not  be  able  to  be  rigorous 
testers  because  of  "leniency  bias."  If  examiners  are  not  rewarded  for  rigorous 
testing  (they  may  even  be  punished  if  too  many  trainees  fail),  they  will  not 
be  strict  in  their  interpretation  of  their  role.  If  trainees  are  getting  help 
from  examiners,  you  will  have  to  find  out  why  the  examiners  are  doing  this. 

If  they  do  not  realize  what  they  are  doing,  they  need  training  in  test 
administration.  If  the  problem  lies  with  the  attitudes  of  training 
management,  you  can  try  to  solve  this  at  the  next  higher  level  in  the  chain 
of  command  if  that  is  feasible  from  your  position.  If  you  work  for  the 
training  manager,  you  may  be  stuck  on  this  one.  You  would  tailor  the 
correction  to  the  reason. 


102 


1.  Is  the  TEST  FORMAT  appropriate 
for  the  OBJECTIVE? 

2.  Are  there  test  items  for  the  TLO 
or  all  of  its  critical  parts/LOs9 
(See  job  aid  for  critical  parts.) 


3.  Is  there  a  test  item  for  each 
critical  part  of  each  LO? 

(See  job  aid  for  critical  parts.) 


4.  Does  the  TASK  LEVEL  of  the 
test  item  match  the  TASK  LEVEL 
of  its  OBJECTIVE? 

5.  Does  the  content  of  the  test  item 
match  the  content  of  its 
OBJECTIVE? 

6.  Do  the  CONDITIONS  of  the  test 
item  match  the  CONDITIONS 

of  its  OBJECTIVE? 

7.  Do  the  STANDARDS  of  the  test 
item  match  the  STANDARDS  of 
its  OBJECTIVE? 

8.  For  true-false,  multiple  choice, 
and  matching  items  is  only  one 
answer  correct? 

9.  For  short  answer,  fill-in,  listing, 
and  performance  items  are  all 
acceptable  answers  in  the  answer 
key? 

10.  Does  the  test  item  provide 
opportunities  for  COMMON 
ERRORS  to  be  made? 

11.  Is  the  language  of  the  test  item 
easy  for  students  to  understand? 


12.  Is  the  test  item  different  from 
previous  PRACTICE  and 
EXAMPLES?  (USE-CONCEPT, 
USE-RULE,  or 
USE-PRINCIPLE  only) 


(See  table  in  job  aid.) 


1  =  Test  items  for  TLO  or  all 

parts/LOs 

2  =  No  test  items  for  TLO  and 

for  some  parts/LOs 

3  =  No  test  items  for  TLO  and 

for  most  parts/LOs 

1  =  Items  for  all  parts 

2  =  Items  for  many,  but  not  all  parts 

3  =  Items  for  only  a  few  parts  or 

for  no  parts 

(See  table  in  job  aid.) 


1  =  Same 

2  =  Slightly  different 

3  =  Very  different 

1  =  Exact  match 

2  =  Minor  mismatch 

3  =  Severe  mismatch 

1  =  Exact  match 

2  =  Minor  mismatch 

3  =  Severe  mismatch 

1  =  Only  one  answer  is  correct 
3  =  More  than  one  answer  can  be 
correct 

1  =  All  correct  answers  are  in 
answer  key 

3  =  Some  correct  answers  are  not 
in  answer  key 

1  =  Yes 
3  =  No 


1  =  Easy 

2  =  Somewhat  difficult 

3  =  Very  difficult 

1  =  Different 

2  =  Presented  before,  USE-UNAIDED 

3  =  Presented  before,  USE-AIDED 


Is  the  answer  to  the  test  item 
given  away  by  other  item(s)? 


14.  Is  the  answer  to  the  test  item 
dependent  on  answering  previous 
item(s)  correctly? 


15.  Are  sketches  and  diagrams  used 
in  the  test  item  easy  to 
understand? 

16.  Is  the  test  item  tricky  or 
misleading? 


17.  Is  the  test  item  well  constructed? 
(See  job  aid  for  criteria  list  for 
the  test  format  used.) 

18.  When  performance  steps  are 
scored,  does  the  instructor  use 
a  checklist? 


19.  Is  each  correct  answer  position 
used  about  the  same  number  of 
times?  (true-falsd,  multiple 
choice,  or  matching  items  only) 

20.  Are  specific  patterns  of  correct 
answer  positions  repeated  across 
test  items  or  are  single  positions 
repeated  in  blocks?  (true-false, 
multiple  choice,  or  matching 
items  only) 

21.  Are  test  administration 
directions  complete? 


22.  Do  instructors  follow  the 

directions  when  administering 
the  test? 


23.  Are  adequate  test  instructions 
provided  to  the  student? 


Answer  not  given  away 
Other  items  give  clues 
Answer  can  be  found  in  other 
item(s) 

Answer  not  dependent  on 
other  items 

Previous  items  must  be  correctly 
answered 

Easy  to  understand 
Somewhat  confusing 
Very  confusing 

Not  misleading 
Somewhat  misleading 
Very  misleading 

Meets  all  criteria 

Deficient  on  noncritical  criteria 

Deficient  on  critical  criteria 

Fills  in  completely 
Uses  as  a  reference  or  fills 
in  partially 
Does  not  use 


No  patterns  easily  seen 
Patterns  can  be  seen 


Directions  are  complete 
Directions  provided,  but 
incomplete  or  unclear 
Directions  are  not  provided 


Some  variations  from  directions 
Significant  variations  from 
directions 

Yes 

Instructions  provided,  but 
unclear 

No  instructions  provided 
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24.  Does  the  FINAL  TEST  integrate 
tasks  as  they  are  integrated  in 
the  "real  world?" 

25.  Are  tasks  and  task  steps  tested 
in  the  same  sequence  as  they  are 
performed  in  the  "real  world?" 

26.  In  the  test  free  of  external  cues 
or  help? 

27.  Are  motivational  techniques 
employed? 

28.  Is  the  trainee  attitude  positive? 


29.  Are  course  ENTRY  SKILLS 
reviewed? 


30.  Is  mastery  of  prerequisite  skills 
verified  prior  to  new  instructions7 

31.  Are  OBJECTIVES  presented  to 
the  student? 

32.  Are  the  basic  PRESENTATION 
COMPONENTS  present? 

33.  Are  STATEMENTS  complete?  • 


34.  Are  STATEMENTS  for  CONCEPTS, 
PROCEDURES,  or  RULES 
adequate?  (See  job  aid  criteria.) 

35.  Does  STATEMENT  HELP  provide 
sufficient  explanation? 


36.  Does  training  include  instruction 
on  the  use  of  required  job 
performance  aids? 

37.  Are  EXAMPLES  and 
NON-EXAMPLES  adequate? 


1  =  Yes 

2  =  Partially 

3  =  No,  tasks  are  tested  separately 

1  =  Yes 

2  =  Slightly  out  of  sequence 

3  =  Very  different  sequence 

1  =  Yes 

2  =  Hints  given 

3  =  Answers  are  given  away 

1  =  Yes 

3  =  No 

1  =  Positive 

2  =  Indifferent 

3  =  Hostile  or  frustrated 

1  =  Review  with  practice 

2  =  Review  with  no  practice 

3  =  No  review 

1  =  Yes 
3  =  No 

1  =  Yes 
3  =  No 

(See  guidance  and  tables  in 
handbook.) 

1  =  STATEMENT  complete 

2  =  Few  parts  missing 

3  =  Many  parts  missing 

1  =  Completely  adequate 
3  =  Some  or  all  features  omitted 


1  =  Help  provides  sufficient 

explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  =  Yes 
3  =  No 


1  =  Yes 
3  =  No 


38.  Is  EXAMPLE  HELP  adequate? 


39.  Are  EXAMPLES  sequences  from 
easy  to  hard?  (CONCEPTS  only) 

40.  Are  there  enough  EXAMPLES? 

(See  job  aid  for  criteria.) 

41.  Are  NON-EXAMPLES  included? 
(CONCEPTS  only) 

42.  Do  DEMONSTRATIONS  show 
how  to  correct /avoid  common 
errors? 

43.  Are  steps  in  a  DEMONSTRATION 
the  appropriate  size?  (See  job 
aid.) 

44.  Are  tasks  and  task  steps 
DEMONSTRATED  in  the  same 
sequences  as  they  are  performed 
in  the  "real  world?" 

45.  Are  memory  aids  used? 

(PRACTICE  REMEMBERING 
only) 

46.  Does  each  PRACTICE 
REMEMBERING  have  the  same 
content  and  format  as  the  test 
item? 

47.  Are  PRACTICE  USING  items 
sequenced  from  easy  to  hard? 

48.  Do  PRACTICE  USING  items 
provide  opportunities  for 
COMMON  ERRORS  to  be  made? 

49.  Are  PRACTICE  items  different 
from  EXAMPLES?  (USE-CONCEPT, 
USE-RULE,  or  USE-PRINCIPLE 
only) 

50.  Does  PRACTICE  USING  integrate 
tasks  as  they  are  integrated  in  the 
"real  world?" 
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1  =  Help  provides  sufficient  \ 

explanation  j 

2  =  Help  gives  insufficient  : 

explanation  ] 

3  =  Help  is  confusing  i 
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1  =  Yes  i 

3  =  No  ; 

1  =  Yes  ! 

3  =  No  | 

1 

1  =  Yes  > 

3  =  No 

1  =  Yes 

3  =  No 


1  =  Yes 

2  =  Step  size  is  too  small 

3  =  Step  size  is  too  large 

1  =  Yes 

2  =  Slightly  out  of  sequence 

3  =  Very  different  sequence 


1  =  Used 
3  =  Not  Used 


1  =  Same 

2  =  Same  content,  different 

format 

3  =  Different  content 

1  =  Yes 
3  =  No 

1  =  Yes 
3  =  No 


1  =  Different 
3  =  Presented  before 


1  =  Yes 

2  =  Partially 

3  =  No,  tasks  are  practiced 

separately 


51.  Are  job  performance  aids 
(JPAs)  usable?  (See  criteria 
in  job  aid.) 

52.  Do  all  students  use  the  job 
performance  aid  (JPA)? 


53.  Does  the  TASK  LEVEL  of  the 
PRACTICE  item  match  that  of  the 
test  item(s)? 

54.  Does  the  CONTENT  TYPE  of 

the  PRACTICE  item  match  that  of 
the  test  item(s)? 

55.  Does  the  FORMAT  of  the 
PRACTICE  item  match  that  of 
the  test  item(s)? 

56.  Do  the  CONDITIONS  of  each 
FINAL  PRACTICE  item  match 
those  of  the  test  item(s)? 

57.  Do  the  STANDARD  of  each 
FINAL  PRACTICE  item  match 
those  of  the  test  item(s)? 

58.  Is  FINAL  PRACTICE  free  of 
external  cues  or  help? 

59.  Are  there  PRACTICE  items  for 
each  TLO  or  all  of  its  critical 
parts/LOs? 


60.  Is  there  a  PRACTICE  items  for 
each  critical  part  of  each  LO? 
(See  job  t  *  for  critical  parts.) 


61.  Do  all  students  PRACTICE? 


1  =  Easy  to  use 

2  =  Hard  to  use 

3  =  Unusable 

1  =  Yes 

2  =  Up  to  20%  do  not  use  3PA 

3  =  More  than  20%  do  not  use 

3PA 

1  =  Yes 

(2  and  3  —  See  table  in  job  aid.) 


1  -  Yes 
3  =  No 


1  =  Yes 

(2  and  3  —  See  table  in  job  aid.) 


1  =  Yes 

2  =  Slightly  different 

3  =  Very  different 

1  =  Yes 

2  =  Slightly  different 

3  =  Very  different 

1  =  Yes 

2  =  Hints  given 

3  =  Answers  are  given  away 

1  =  PRACTICE  items  for  the 

TLO  or  all  parts/LOs 

2  =  No  PRACTICE  items  for  the 

TLO  and  for  some  parts/LOs 

3  =  No  PRACTICE  items  for  the 

TLO  and  for  most  parts/LOs 

1  =  PRACTICE  for  all  parts 

2  =  PRACTICE  for  many,  but  not 

all  parts 

3  =  PRACTICE  for  only  a  few 

parts  or  for  no  parts 

1  =  Yes 

2  =  Up  to  20%  of  students  do  not 

PRACTICE 

3  =  More  than  20%  of  the  students 

do  not  PRACTICE 


62.  Do  all  students  meet  the 

required  STANDARDS  in  FINAL 
PRACTICE'7 


63.  Is  FEEDBACK  provided  for 
PRACTICE-7 


64.  Is  FEEDBACK  HELP  adequate? 


65.  Is  TEAM  PRACTICE  provided? 


66.  Are  TEAM  PRACTICE 
CONDITIONS  the  same  as  (or  as 
close  as  possible  to)  those  of 
the  real  task? 

67.  Is  TEAM  PRACTICE  FEEDBACK 
provided? 


68.  Is  FEEDBACK  HELP  for  TEAM 
PRACTICE  adequate? 


69.  Are  all  PRESENTATION 
COMPONENTS  separated  and 
identified? 

70.  Is  the  technical  quality  of  written 
or  spoken  material  adequate? 

(See  job  aid  for  criteria.  Make 
notes  on  specific  problems.) 

71.  Is  the  wording  of  written  or 
spoken  material  easy  for  the 
students  to  understand? 


72.  Is  the  instructor's  presentation 

or  the  narration  easy  to  listen  to? 


1  =  Yes 

2  =  Up  to  20%  of  students  do  not 

3  =  More  than  20%  of  students 

do  not 

1  =  FEEDBACK  HELP  is  given 

2  =  Correct  answer  only  is  given 

3  =  No  feedback  is  given 

1  =  Help  gives  enough 

explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  =  Yes 
3  =  No 

1  =  Yes 

2  =  Slightly  different 

3  =  Very  different 


1  =  FEEDBACK  HELP  is  given 

2  =  Success/Failure  feedback 

only  is  given 

3  =  No  feedback  is  given 

1  =  Help  gives  enough  explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  =  Yes 

2  =  Some  are  not 

3  =  Most  or  all  are  not 

1  =  Most  criteria  met 

2  =  Several  criteria  not  met 

3  =  Few  criteria  met 


1  =  Yes,  few  hard  words  and 

long  sentences 

2  =  Some  hard  words  and  long 

sentences 

3  =  Many  hard  words  and  long 

sentences 

1  =  Yes 

2  =  Dull  and  monotonous 

3  =  Hard  to  listen  to 
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73.  Is  the  instructor's  presentation 
or  the  narration  supported  by 
visuals? 

74.  Are  visuals  easily  understood0 


75.  Are  the  OBJECTIVES  (TLOs  and 
LOs)  within  each  LESSON 
sequenced  properly? 

(Prerequisites  taught  first.) 

76.  Are  the  LESSONS  sequenced 
properly  within  the  course? 

77.  Are  the  media  appropriate  for  the 
objectives7  (See  table  in  job  aid.) 


78.  Can  the  media  used  provide  all 
necesary  stimuli? 

79.  Are  the  course  administration 
directions  complete? 

80.  Do  course  administration 
directions  make  re'alistic  demands 
of  students  and  instructors? 

81.  Is  the  instructor/trainee  ratio 
such  that  all  students  can  see, 
hear,  and  receive  feedback? 


82.  Does  the  instructor  follow  the 
methods  in  the  Instructor  Guide? 


83.  Does  the  instructor  teach  all  of 
the  content  in  the  LESSON 
materials? 


84.  Did  the  instructor  limit  his 
teaching  to  t.ie  content  in  the 
LESSON  materials? 


1  =  Completely 

2  =  Partially 

3  =  Not  at  all 

1  =  Yes 

2  =  Understandable  with  effort 

3  =  Very  hard  to  understand 

1  =  Yes 
3  =  No 


1  =  Yes 
3  =  No 

1  =  Yes 

3  =  No  (note  key  words, 
underlined  in  table,  on 
worksheet.) 

1  =  Yes 
3  =  No 

1  =  Yes 

2  =  Partially  incomplete 

3  =  Incomplete  or  non-existent 

1  =  All  demands  are  realistic 
3  =  Some  demands  are  unrealistic 
(Note  what  they  are.) 

1  =  Yes 

2  =  A  few  students  cannot  see, 

hear,  and  receive  feedback 

3  =  Many  students  cannot  see, 

hear,  and  receive  feedback 

1  =  Yes 

2  =  Follows  to  some  extent 

3  =  Follows  very  little  or 

not  at  all 

1  =  Yes 

2  =  Much  of  the  content 

3  =  Very  little  of  the  content 

(If  2  or  3,  note  what  was 
left  out.) 

1  =  Yes 

3  =  No  (Please  note  what  other 
things  he  taught.) 


85.  Is  there  enough  space  for  all 
of  the  trainees? 


86.  Is  instruction  free  of 
distractions? 


87.  Is  the  lighting  appropriate  for 
the  training  situation? 


88.  Is  the  temperature  appropriate 
for  the  training  situation? 


89.  Is  the  instructor's  attitude 
positive? 

90.  Are  frequent  breaks  provided? 
(5-10  minute  breaks  every  hour) 


91.  Is  the  speed  of  presentation 
appropriate? 


92.  Was  the  allotted  training  time 
too  long  or  too  short? 

93.  Does  the  training  device/ 
equipment  used  in  training 
function  properly? 


1  =  Yes 

2  =  A  little  crowded 

3  =  Very  cramped  or  some 

students  can't  fit  in  the 
space  at  all 

1  =  Yes 

2  =  Distractions  are  annoying 

3  =  Distractions  seriously 

interfere  with  the 
instruction 

1  =  Yes 

2  =  Students  have  trouble 

reading  or  seeing  displays 
ic.  equipment 

3  =  Students  cannot  read  or  see 

displays  &  equipment 

1  =  Yes 

2  =  Temperature  makes  students 

uncomfortable 

3  =  Temperature  seriously 

interferes  with  learning 

1  =  Yes 
3  =  No 

1  =  Yes 

2  =  Breaks  too  short  or 

infrequent 

3  =  Breaks  not  provided 

1  =  Yes 

2  =  Too  slow 

3  =  Too  fast 

1  =  Appropriate  length 

2  =  Too  long 

3  =  Too  short 

1  =  yes 

2  =  Minor  malfunctions,  little 

change  from  intended  task 
performance 

3  =  Major  malfunctions,  substantial 

change  from  intended  task 
performance 


94.  Is  there  anything  else  unusual  about  the  LESSON  materials,  or  do  any  other 
critical  incidents  occur  during  training  that  would  interfere  with  learning? 
(Describe  each  one  below.  Rating  =  3) 
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MEDIA  DEFINITIONS 


Appendix  B 
MEDIA  DEFINITIONS 


Audio.  Equipment  which  delivers  a  pre-recorded  audio  message,  which  may  be 
amplified  for  groups.  The  device  involved  may  also  have  recording  capability. 
Audio  systems  include  audio  tape  systems,  either  cassette  or  reel-to-reel; 
stylus-  or  laser-read  disc  systems;  and  computer-generated  audio. 

Chart.  A  flat  surface  for  presenting  displays  to  a  class,  containing  messages 
in  print,  handwriting,  drawings,  pictures,  graphs,  or  other  forms.  A 
blackboard  is  included  in  the  chart  category. 

Computer.  The  computer  displays  printed  words,  symbols,  and  diagrams,  and 
may  accept  keyboard,  light  pen,  or  touch  inputs  as  student  responses. 

Suitably  programmed,  it  can  provide  instruction  adapted  to  the  individual 
student's  state  of  knowledge,  display  corrective  feedback,  and  perform  other 
adaptive  and  interactive  operations.  New  varieties  of  instructional  use  are 
foreseen  as  auditory  inputs  and  outputs  become  available. 

Filmstrip.  Projection  of  pictures  which  are  sequentially  printed  on  a  strip  of 
film.  Such  projection  may  or  may  not  be  synchronized  with  an  audio  message 
from  a  tape  player.  The  function  of  this  equipment  is  the  same  as  that  of 
slide  projection,  with  and  without  synchronized  audio.  It  is,  however,  a 
separate  medium,  because  it  requires  different  materials  and  equipment  and 
cannot  be  readily  re-sequenced  as  slides  can. 

Instructor.  As  a  source  of  instruction,  the  most  versatile  medium  is  an 
instructor.  Four  important  functions  of  an  instructor  are:  (a)  delivery  of 
audio  messages,  (b)  directing  individual  student  attention  to  particular  features 
of  a  display;  and  (c)  human  modeling,  by  serving  as  a  model  for  choices  of 
personal  action;  and  (d)  providing  corrective  feedback  to  the  individual 
student.  Types  of  individuals  that  could  fall  into  this  category  would  be 
instructors,  who  can  perform  any  of  these  functions,  and  assistant  instructors, 
who  typically  perform  limited  functions,  such  as  providing  feedback. 

Interactive  TV.  This  medium  combines  a  microcomputer  with  videodisc  or 
videotape  playback  equipment.  An  interactive  TV  program,  with  sound,  can 
become  a  highly  useful  "teaching  machine"  since  its  branching  capabilities  can 
provide  feedback  tailored  to  the  individual  student.  It  is  different  from  a 
computer  in  that  it  can  display  motion-visuals  and  realistic  still-visuals. 

Large  Equipment.  Operating  equipment  that  is  large  and  non-portable,  such 
as  a  helicopter,  truck,  or  artillery  piece. 

Motion  Picture.  A  film  projection  system  displaying  moving  pictures  on  a 
screen,  often  accompanied  by  a  synchronized  audio  track.  Some  systems 
permit  large  screen  projection  to  large  groups.  Systems  in  this  medium 
include  8mm  projector  systems  (either  regular  or  super  8),  as  well  as  16mm 
and  35mm  systems.  Included  in  this  category  are  both  sound  and  silent 
projection  systems  although  the  latter  is  infrequently  used. 


9. 


Overhead  Projection.  Projection  of  transparencies  (about  8  by  10  inches) 
displaying  print,  pictures,  or  diagrams  onto  a  nearby  screen.  Overhead 
projection  and  the  changing  of  transparencies  may  be  done  by  the  instructor 
at  the  front  of  a  classroom  or  by  using  rear-screen  projection.  This  category 
also  includes  the  opaque  projector. 

10.  Portable  Equipment.  A  unitary  device  (rifle,  protective  mask),  or  a 
component  of  a  real  system  (dial,  carburetor),  which  is  small  enough  to  be 
readily  lifted  and  transported.  Portable  equipment  is  often  the  medium  of 
choice  whenever  direct  practice  of  task  performance  is  possible  and  desirable. 
Such  equipment  may  sometimes  have  a  specially  built-in  mechanism  to  provide 
corrective  feedback  to  students,  and  thus  may  function  in  the  manner  of  a 
training  device. 

11.  Printed  Text.  Pages,  cards,  or  other  surfaces  containing  meaningful  materials 
in  printed  form,  for  use  by  individual  students.  It  is  always  possible  for 
printed  text  to  contain  printed  pictures  or  diagrams.  Microfiche,  microfilm, 
and  other  such  reduced  print  media  will  be  included  in  this  category,  even 
though  they  require  additional  equipment  for  their  use. 

12.  Programmed  Text.  This  medium  is  the  basic  form  of  programmed  instruction, 
using  printed  cards  or  pages,  usually  collected  into  a  booklet.  A  printed  text 
(see  item  11)  takes  on  a  programmed  form  when  it  requires  frequent 
responses  and  provides  feedback. 

13.  Simulator.  Typically,  equipment  used  for  training  or  proficiency  maintenance 
which  reproduces  many  of  the  operating  characteristics  of  the  real  equipment. 
An  example  is  an  aircraft  flight  simulator. 

14.  Slides.  A  medium  without  audio  which  projects  still  pictures  from  small 
slides  onto  a  variety  of  possible  screens,  ranging  from  inches  to  yards  across. 

15.  Slide/Tape.  Projection  of  still  pictures  from  small  slides,  accompanied  by 
synchronoized  audio-tape  messages.  This  category  may  include  a  combined 
slide  projector /audio  tape  system  with  automatic  cueing  or  separate  slide  and 
audio  tape  machines  with  synchronization  provided  through  direct  human 
assistance.  Slide/tape  systems  may  involve  more  than  one  projector. 

Although  the  Bessler  Cue-See  can  provide  motion  visuals,  it  is  included  in  the 
slide/tape  category  because  its  typical  use  is  closer  to  slide/tape  than  it  is  to 
motion  pictures. 

16.  Training  Aid.  A  surface  layout,  model,  or  mockup,  providing  a  display  of 
parts  and  processes  of  a  system  on  which  instruction  is  being  given.  The 
simplest  form  of  a  training  aid  is  a  chart.  However,  training  aids  may 
display  part  of  real  equipment  (rather  than  pictures),  and  even  provide 
dynamic  views  of  processes  (as  in  a  working  model  of  a  hydraulic  system). 

In  contrast  to  training  device  (see  item  17),  a  training  aid  makes  no  specific 
provision  for  corrective  feedback  to  the  student. 

17.  Training  Device.  Equipment  on  which  mental  skills  procedures  or  motor  skills 
can  be  practiced,  and  which  provide  corrective  feedback  to  the  student.  (A 
training  aid  does  not  automatically  provide  corrective  feedback.)  A  training 
device  has  its  own  operating  principles  and  does  not  attempt  to  reproduce 
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those  of  the  real  equipment,  as  does  a  simulator.  An  example  of  a  training 
device  is  a  Link  trainer. 

18.  TV  Cassette.  A  device  which  makes  possible  the  display  of  a  pre-recorded 
program  (picture  and  sound)  on  a  TV  receiver.  The  device  may  also  have  a 
recording  capability.  Delivery  systems  covered  by  this  category  include  reel- 
to-reel  videotape  systems,  videocassette  systems,  and  videodisc  playback 
systems. 
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